News: 1588575730

Well, whoever was doing the testing earned their pay packet! I wonder how the tests were devised and how much of their content was based on 'gut feelings', beyond a few standard sets?

Experience is everything here.

I did a recent project and the customer was insistent that UAT should be prescriptively scripted.

It took more than a few wasted hours in meetings to get them to finally see that doing that makes users do the tests in front of them and only the tests. They won't deviate and nor will they report any aberrant behaviour that isn't part of the scripted tests.

These are people we've picked who have years of experience using the software daily. Let them do their jobs as they normally would and report the issues they come up with.

Other tests along the way can be more formalised.

Users have a habit of using software in ways the designers never thought of.

Many years ago I had the absolute pleasure of working with two testers Cheryl and Caroline. Outside of the office they were great company and we all (developers and testers) got on like a house on fire. Inside the office they had PMT ratcheted up about 1000%, which was absolutely brilliant for the job!

The bug report I found the most amusing/terrifying went along the lines of :

System - some employee pay and benefits system

Problem - cannot enter minus one for number of children of employee

Rational - Employee has divorced and their partner now has sole custody

Fellow developers can get even more creative...

As much as they might try, developers are never as creative at fucking systems as end-users. And end-users don't need to try to get creative, they just fuck systems up and don't even know how or why

"Users have a habit of using software in ways the designers never thought of."

Yeah, the bastards.

Bloody users, fouling up my elegant software with their 'requirements' and 'needs'.

They won't deviate and nor will they report any aberrant behaviour that isn't part of the scripted tests.

Yep, it only tests that it does what it is supposed to. Testing that it does not do what it isn't supposed to is far more useful (and far more difficult).

Yeah back in the day when I moonlighted as a tester when work had nothing else for me to do, the fun part was going outside the prescribed the test cases and getting things to fail.

I visited a Unisys plant once that was close to releasing a new version of software to configure a communications controller. There were signs all round the plant asking employees to spend a given day trying to break the software in any way they could.

I stll think a formal script is necessary, but maybe it isn't sufficient.

"I did a recent project and the customer was insistent that UAT should be prescriptively scripted."

That's what automated tests are for, and it's valuable... You want to make sure that regressions don't occur, but that's only one string of a test, the more creative testing is far more valuable.

But the output of the "creative" testers will inform automated tests for future versions, but the creative ways to break a system are always the most interesting - and the ones that no sane person would prescribe in a test. They're the things that an experienced tester will go ... Something odd about how that window loaded in, what if I just...

I always find a bit of both works best, as you will get some users query why do I need to do such and such process this way.

Often best to get the underlings to do the testing in said departments and ensure different people repeat the tests.

"I did a recent project and the customer was insistent that UAT should be prescriptively scripted."

Obviously somebody who'd never been told that no battle plan survives first contact with the enemy.

The tester is described as being an engineer and probably was. Therefore someone methodical and used to the logic of testing things such as what if x fails? what if y fails? etc.

My father was an engineer. My attempts at self designed Meccano models had a harsh, but constructive critic.

He worked for NZ Rail and part of his job was testing and certifiying machinery such as cranes. He had overalls for when he needed to clamber over things. He said that was something he enjoyed. He ws the sort of guy who would need to eyeball and knock every pneumatic joint, assess clearances, look for cracks etc. How might this thing fail?

In NZ whenever there's an earthquake nothing runs while people on little motorised tractors trundle over the lines checking nothing has fallen in the tunnels, the bridge connections are intact (some of the bridges are designed to break the line if the quake is big enough so the bridge can sway independently). i expect these days they also use drones.

Testing to destruction as well. That's what many engineers do, when they don't like something. Turn the voltage up to 11 and see what happens.

Sticking with rail, but skipping across the water, one company I worked at helped with the Australian VFT (Very Fast Train - very pragmatic naming in Australia, I always joked it would be replaced by the FFT, I'll leave you to work that out!). A colleague was at the launch press conference and allegedly, someone from the press asked what they would do if their was a kangaroo on the track (we are talking about a train doing in excess of 120mph).

The managers and consultants all looked at each other, shrugs were given, then some bright spark said "turn on the wipers?" I guess they hadn't thought about every scenario for the press conference, although I'm sure the engineers had tested up to camel/buffalo...

When I was a youth, they introduced the XPT express passenger trains, relatively high speed trains on rural lines. From memory, they were wiping out cars at level crossings every month or 2.

The XPT was derived from British Rail's HST, with downrated engines, uprated suspension and a reduced top speed. As I recall, there was only 1 XPT crash at a level crossing, the notorious crossing on the Olympic Highway at Gerogery in NSW. In the run up to he XPT crash there had been numerous other crashes at the crossing, which ws eventually replaced with an overbridge called "Five Mates Crossing" after the five lads in the car who died as a result of the XPT collision.

I take it the one after a FFT would just be called "Strewth"?

"what they would do if their was a kangaroo on the track"

Correct reply - nothing, it's the kangaroo that has the problem.

Whoever it was, he was never employed at Boeing, apparently.

But honestly, this is not a tale of a major blunder saved in extremis . This is just a normal development cycle. Developer codes, tester tests, results come back and the cycle starts again until the code is approved for production.

That is exactly what happened.

Many years ago I was a data recovery engineer. I was also responsible for writing and maintaining our data recovery tools. Every now and again the hardware engineers would whine about the time it was taking to image hard drives. Sometimes I'd offer sympathy and agree to investigate. It was a way to avoid real work since I could usually blame the network or the (NetWare) servers. Other times I'd just shrug it off and do something more important.

Anyway this one time the engineers were adamant things had suddenly taken a turn for the worse. So there was a department-wide investigation. The network was checked and found to be fine. The servers seemed okay as well. Eventually we gave up and forgot about it for a week or two. Then while I was investigating another problem I happened across the main I/O loop for our disk imager. And there I found a debug statement, left in while investigating our 'non-BIOS' disk reading. This was code that used [1]ATA to talk directly to the disk. We needed that because sometimes the BIOS just couldn't handle the state the drive was in and occasionally we took in a drive so large that [2]the BIOS couldn't access it all . Anyway this code could be a bit temperamental and it often came down to timing.

Hence the debug statement I'd inadvertently left in. The I/O loop read 64kiB of data then wrote it out. Then it waited 10ms before going round for more data . I toyed with the idea of pretending I'd discovered a hitherto unknown way to improve I/O but we were a friendly team so I 'fessed up. Sometimes it does you good to laugh even if it makes you look like a chump.

[1] https://en.wikipedia.org/wiki/Parallel_ATA

[2] https://en.wikipedia.org/wiki/Logical_block_addressing#Enhanced_BIOS

I have never seen an engineer in a purple haze , but I have seen them turn a whiter shade of pale .

I was part of a QA team testing an office machine that was about to be launched. We discovered it was simple to run the machine with the safety covers open. The chief engineer of the project said this was "absolutely impossible" and that we had made a mistake.

We gathered them around the test machine and opened all the safety covers - the machine stopped and the engineer smiled.

We left the doors open and cycled the power - the machine started and the engineer went white as a sheet.

My wife was writing docs for a product, and filed a bug that it didn't work on Wednesdays. A somewhat sceptical development team agreed to humour her by looking at the problem, and found a day-of-week variable that was only 8 characters in size ...

It was drummed into me, when I started analysis and programming, that 60% - 70% of a projects time is testing.

You received the spec and wrote your test plan, only then did you start on the code and the test harness. You coded to the spec and test plan, you didn't write the test plan to fit the code.

Then the code went to system testing, who only ever saw the spec, never the code, so their test plans were spec based.

When that was complete, the test was signed off and the code put into pre-production, where the customer had their acceptance testers. Once they had signed off the code as correct, it could be moved to production...

Nowadays, it often seems to be people throwing links between frameworks together, then throwing the whole lot "out there" and waiting for the screams.

Nowadays, it often seems to be people throwing links between frameworks together, then throwing the whole lot "out there" and waiting for the screams.

It's called "Agile".

It's called " fr Agile".

FTFY

It's called "Windows 10".

FTF FY.

Absolutely

Guaranteed

Indubitably

Lamentable

Errors

In my defence when you're writing data recovery tools for the engineers that are using them you can't always afford time for thorough testing. Most customers want their data back yesterday so you bodge up a fix for whatever is blocking that particular recovery and worry about the code later. But then there's a dual nature of the job where sometimes you had to drop the programming you were in the middle of and actually recover some data. In the early days we even had to answer the phone and give quotations to customers. Can you say 'context switch'?

I think on balance looking back over the 15 years I did that (15 years during which the main tools were ported from DOS, to Win16 then to Win32 using both Pascal and C++) we did a damn good job.

Happy days but history now.

This sounds lovely. I must be in the wrong industry :-(

I'm old enough to remember when the tenner came in a little plastic cassette, which you were supposed to return in the slot provided.

The street around the machine used to be covered with them.

You also had to go into the bank branch the next day to get your "cash cards" back.

We gave our acceptance testers specific instructions. "Try and break it!"

I have had a few "blue smoke moments" when testing prototypes. One of the worst was when testing a power supply for endurance at full load. It took hours for the thing to warm up, then "boomf!". I took the lid off, and there was a bit of a crater in the PCB. A few power devices had died. It looked like there had been a short, perhaps due to bad soldering. I thought this because our regular technician was busy at the time, so the production engineer did the soldering. The board was patched, and the dead parts replaced. Switch on, load up, and wait. A few hours later -- "boomf!". OK, this was not bad assembly. Something was overheating. I eventually tracked it down to a device that needed a higher voltage rating, in order to reduce leakage current that caused thermal runaway. I should have known about this, but the data that would have warned me was somewhat buried in the component datasheet. "Every day is a school-day", as my boss said at the time.

This faffing about with repeated lengthy tests delayed the product launch somewhat, but you have to consider what may have happened otherwise. Say we shipped hundreds of PSUs in the winter, and they did not overheat at that time. Come the summer, and they start going "boomf!" all over the country. Multiple callouts at our expense, and all profits wiped out.

While "Boomf" might be appropriate, I tend to find that mains-electrical stuff goes "Spop!"

Weirdest noise I got was from a soldering iron that got wet.

Dried it out and turned it on, although it was probably dead.

It gave a soft "ehurgh" coughing noise before committing seppuku across the table.

We had one tester who had some sayings

- There are two ways of reporting testing. 1) "We have run all of the tests, and on a good day with the wind behind us, they all passed" or 2) "No matter what we did - we could not break it". Management want 2) but will take 1) and run.

- The point of test is to change the product until the all the tests pass.

- Automation is great - but eventually the product is fixed and all the test run. You then change the automation - slow it down, speed it up, test different things.

- If you haven't broken it - you havent pushed it hard enough

My experience of catastrophe prevented, was testing on the biggest z/OS mainframes configured for High Availability, running CICS, IMS, and MQ. I ramped up the transaction rate and within minutes one mainframe had crashed. But this was configured for HA... so the work seamlessly moved to the next mainframe -for a minute, and this then crashed - and the work moved to the next mainframe - which also died.

No one else had this problem not event the stress team - I thought it was my dodgy monitoring code.

It turned out that under extreme load some code was not freeing "system storage" because someone had copied only 99% of a fix from somewhere else. They added the one line of code saying "this block is free" and it worked, and I could not get it to fail.

We had a HA system which worked fine. Then one day, the primary failed and everything jumped over to the secondary. Which promptly failed and everything failed back to the primary (which was just about getting back on its feet) Which failed, etc. , etc..

It turned out one client got into a very weird state and sent command packets to the servers which the servers couldn't handle. Once we managed to get some logs and isolate the client the system became a lot more stable.

I went to university in 1972

I had a cash card from Natwest Bank

It was a small plastic punch card. You put in the slot with a PIN and got £10

They then posted it back.

By doing this and getting the maximum on a cheque at the counter with my guarantee card,I think £25, I could pay a terms rent.

One of my friends had a card like that. We realised that it looked just like 2" cut off the end of a standard punched card, but he wouldn't let us borrow it to see if we could make a copy...

My reaction to the "25p a pint" was to think they were being ripped off.

When I was at Uni, 75 - 78, a group of 5 of us used to have a regular Friday evening at the pub. Each of us paid £1 for a round - 4 of us bought rounds of beer (Hardy and Hansons Kimberley Ale) and the 5th a round of ham (or cheese and onion) baps. I think we even got 5p change each.

I worked for Burroughs/Unisys and one day an engineer told a story. He'd been called by an irate bank. I won't name them but in those days you were unlikely to find a branch south of Hadrian's Wall. One of their customers had complained that the atm had failed to issue the tenner. A long argument ensued: the engineer defended his machine, explaining that such a situation was impossible.

Years later he was uninstalling the machine in question. Tucked away underneath he found a tenner. Rather than reopen old wounds he secreted it from the premises.

Even the newer ones weren't perfect. Back in the late 80s a colleague was preparing for Christmas shopping, and asked an ATM for £200. It hummed & clicked, and reported that it only had £150 left, and would he take that. He selected "yes", took the cash, and fortunately kept the receipt showing the issue of £150. A few weeks later he got his bank statement, which showed a debit of the originally-requested £200.

It took a lot of arguing with the bank ("that isn't possible, Sir"), but they did eventually agree to refund him the missing £50. We assume that somewhere a bank software engineer had to hastily create a software fix...

I used to use Burroughs/Unisys kit and was on a training course in Milton Keynes. In the evening, we got talking and one of the Unisys instructors regaled us with a story about one of their engineers.

He worked in some remote place and, over a couple of years, every time he visited a customer to repair something, he order 2 of the thing being replace, instead of 1. They said that nobody had noticed and he would have gotten away with it, but he tried to order a new case - no engineer hat ever ordered a new case, so they made an investigation...

I tried the above at the login screen for a brand new super duper system being introduced but used users instead of students, it didn't work but when I used employees it did. Anon because for some strange reason I haven't been hauled up for a disciplinary and I'm keeping it that way.

I did some testing for a retail outfit that was going online at the turn of the Century. It was my first security testing gig and I had to go through the site and see if it was secure.

The first thing I noticed was no SQL Escaping... They didn't seem to think it was important, a quick demonstration of bypassing the customer logon didn't impress them, nor did listing the users table... So for my next trick... DROP DATABASE.

Now, that got their attention!

Posting anon, because we're still in progress.

Working on software that executes real-time commands to a routing system. I represent end-users, the guys who sit and watch to make sure everything goes as planned. Have been asked, "What is your test plan?"

Don't have one, because I never know what the developers are going to break in the next version! This week's update results in 900+ alarms per day due to switches not happening.* We just look at everything to see if something working yesterday is broken today.

*Update installed on Friday. No notice, no e-mail just a flood of alarms all weekend until I logged in to check the system! What fun :/

"*Update installed on Friday. No notice, no e-mail just a flood of alarms all weekend until I logged in to check the system! What fun :/"

Thats just standard.

I guess you work for Virgin Media

We were a customers of a company that sold MPEG hardware many years ago - we would buy the PCI boards put them in PC's and sell the system.

Had some problems where the boards wouldn't work at all - turns out they tested them on old "slower" PCs and we were using them on faster PC's.

There was also their flagship product - several grands worth of PCI that had two notches so that it would fit in either a 3.3V or 5V slot. Except that if you put it in a 3.3V slot it will catch fire..... Their solution - put a warning sticker on it.

There again, they used to ship "diagnostic" software, but if I had a faulty card and wanted to send it back, they would refuse to accept the output of their own test software......

Years ago I was working as a web application developer. I'd been given a legacy (read: poorly coded) application and was tasked with investigating why certain requests were taking so long to respond.

I enabled MySQL's slow query log and had set the threshold for logging as 2 seconds. This means any query that takes longer than 2 seconds to execute gets logged to a file. The application was being used by tens of thousand of users per day. The queries were big and the log files were in the order of GB.

I then went on holiday.

Whilst sat on a beach I got an alert by SMS saying the server's main disk was full. That's odd, I thought, as the disk was about 250 GB.

Not only had I made the error of leaving the query log enabled. Another developer had also written a script which was copying log files to another destination *on the same disk*. The combination of these things filled up the disk very quickly, and the application fell over.

When I returned from my "relaxing" break I was asked why our test plan hadn't worked out. Unfortuantely tests don't always catch things, especially when they weren't reasonably foreseen circumstances at the beginning. But what makes it more interesting is when you combine the sum of those mistakes. On their own these 2 mistakes wouldn't have caused the application to fail. But when combined the result was somewhat different.

When the official testers are driving the vehicle and find something odd just driving between test runs, they told us to "fix it" but it really meant "create a test to duplicate this spurious/random behavior, THEN fix it".

More to that story [1]here . We did manage to find a facility and test methods to consistently replicate the problem. It was mostly a mechanical issue, and some software saved the day. I don't know how many lines, but the core logic was basically three variables in, three value tests, some ANDs and ORs where "true" triggers a single output signal, in parallel with any existing logic that might trigger same signal (just another OR with existing code). The devices and wiring already existed which made it implementation relatively easy -- easier than adding hardware at that stage in development!

And while we're talking testing, I have another relevant post [2]here : good testing kit pays for itself; redundant copies especially.

[1] https://forums.theregister.co.uk/forum/all/2019/03/04/who-me/#c_3730501

[2] https://forums.theregister.co.uk/forum/all/2019/03/11/who-me/#c_3736269