A reader discovered my article from 2018, “It’s a Wonderful Career,” where I said this about software testing jobs: “Get out!” and “Don’t transition to doing test automation.” He asked if I still feel that way now, and the answer is “yes.” Here’s an update on my thoughts.
I’ve been a programmer for most of my life, but a majority of my career has been clustered around software testing. That shifted in 2017 when I took a job as a developer. Since then I’ve flirted with software quality in a 6-month role coaching some test automation folks who were building their programming skills, but otherwise I’ve stayed on the developer track. In my developer roles, I have worked with production code as well as a wide variety of automated tests, plus documentation and release pipeline automation. There have been no testing specialists involved at all.
In my post “Reflections on Boris Beizer” I briefly mentioned how testing as a specialty role has waxed and waned. It was perhaps the 1980s when software the testing specialist became a common role that was distinct from software developers. Fast forward to 2011 when Alberto Savoia gave his famous “Test is Dead” keynote.
I first wrote about the possible decline of the testing role in 2016 in “The black box tester role may be fading away.” I suggested that testing might be turning into low-pay manual piecework (think, Amazon Mechanical Turk), but I don’t see any evidence now that that’s coming true. I mentioned the outsourcing company Rainforest QA. A company by the same name now offers no-code AI-driven test automation instead.
I followed up in “What, us a good example?” where I wrote about companies that aren’t evolving their roles, and yet they’re surviving, but they’re going to have to survive without me. My expectations have evolved.
I know that many people are still gainfully employed as testing and test automation specialists. I can’t fault them for doing what works for them. And I’ll admit that it’s still tempting to go to my comfort zone again and go back to focusing on testing. Maybe I can shed some light on why I’m resisting that temptation. It’s pretty simple, really.
As a developer, my salary has grown tremendously. There are multiple factors involved here, including moving to a different company a few years ago. But the organization I’m working in has no testing specialists, so this opportunity wouldn’t have been available to me if I were applying for a job as a tester. I have a sense there are many more jobs open for developers than for testers out there, especially with a salary that I would accept now, and I’m curious if anyone has any numbers to back that up.
I don’t have any issues with people not respecting my contribution to the product. I don’t have to justify myself as overhead – I have a direct impact on a product that’s making money. And I still do all aspects of testing in addition to developing, maintaining, and releasing products.
In “The Resurrection of Software Testing,” James Whittaker recently described the decline of the testing role as being even more precipitous than I thought. And he also says that it needs to make a comeback now because AI needs testing specialists. He’s organizing meet-ups near Seattle to plot a way forward. I don’t have an opinion on whether AI is going to lead to a resurgence in testing jobs. Instead I’m focusing on an upcoming conference where I’m going to hone my development skills.
And that’s really where I stand on a lot of discussions about testing jobs – no opinion, really. I don’t benefit by convincing testing specialists to change their career path, but I’m happy to share my story if it helps anyone navigate their own career.
One thing I do ponder–there are still so many organizations out there that employ testing specialists, and I might end up working as a developer for one of them some day. How strange that would be if it comes to pass.
[credit for feature image: screen shot from the video of Alberto Savoia’s “Test is Dead” presentation at the 2011 Google Test Automation Conference]
I know you can’t tell from the looks of it, but I’ve been hanging around here a lot lately, working on a post that wants to turn info a book of its own. I’m making good progress getting it tamed into a reasonable length. But first, inspired by LinkedIn posts from James Bach and from Jon Bach and subsequent comments, I want to explore another idea rattling around in my head.
I’m going to talk about three examples where members of a community were accused of groupthink of some sort. In many cases, some people observing the communities say that they see cult-like behavior. I’d prefer not to use the derogatory term “cult” here for a couple of reasons. One, because cult leaders actively encourage groupthink and blind obedience, and I don’t see that happening in these cases (even if their followers are picking up some of these traits). And also, because real cults have done a lot of damage, such as permanently ripping families apart. Let’s not try to equivocate that with what I’m talking about here.
Example 1: I learned a lot from the author and consultant Jerry (Gerald) Weinberg. I am of his followers. People outside his circle often don’t understand the level of devotion that many of his followers exhibited during his life and afterward. Someone even coined a term for it: “Weinborg”, which many of us have adopted for ourselves. (If you don’t get the reference, look up the fictional “Borg” in the Star Trek universe – we have been assimilated).
I attended three of Jerry’s week-long workshops. Every time I’ve been through an intense experiential training, it has been a deeply moving experience. That’s true of Jerry’s workshops, plus other experiential trainings I’ve attended (several Boy Scout training sessions come to mind, for example). Once you’ve recovered, you want more. But you can’t effectively explain why it was so moving to someone who wasn’t there, in fact, for many of them, you can’t give away too many of the details, or you may ruin the experience for someone who attends later.
There are likely many other behaviors among Jerry’s followers that looked odd to outsiders. Perhaps we would invoke his laws by name, like “Rudy’s Rutabaga Rule”. Or we might reference “egoless programming” and point to the book where Jerry wrote about it. We might get ourselves into trouble, though, if we recommended that people follow his ideas without being able to explain them ourselves. “Go read his book” isn’t very persuasive if we can’t give the elevator speech ourselves to show the value in an idea.
Early in my career, a wise leader cautioned me to build opinions of my own rather than constantly quoting the experts. That has been a guiding principle for me ever since, and one that I hope has steered me away from groupthink.
Example 2: James Bach is a consultant and trainer who has influenced a lot of people in the software testing community, along with his business partner. I have learned a lot from James, and I continue to check in with him periodically, though I have never chosen to join his community of close followers. Incidentally, he has also been influenced by Jerry Weinberg.
James has grown a community of people who agree on some common terminology, which streamlines their discussions. It gets interesting, though, when someone uses that terminology outside that community without explaining what it means to them. I remember attending a software quality meetup that advertised nothing indicating that it was associated with James Bach or his courses. But then I heard the organizers and some attendees use terminology that I recognized as originating from James. It’s been several years since the meetings I attended, but I think I remember them presenting other ideas that closely align with what James teaches, not always identifying where they came from or why they recommended them. I vaguely remember that I stood up once or twice and told them that I hadn’t accepted some of those ideas, and I don’t recall the discussion going very far.
If a group has an ideology that they expect participants to adopt as a prerequisite for participating, that’s fine, but it needs to be explicit. Otherwise, they need to be prepared to define their terms and defend their ideas.
Example 3: I participate in the “One of the Three” Slack forum and often listen to its associated podcast created by Brent Jensen and Alan Page. They have spoken out about James Bach and his community a few times. At one point, some participants piled on to some negative comments that seemed to have no basis other than “because Alan and Brent said so.” I called them out for groupthink, not unlike the very thing they were complaining about. Fortunately, I think they got the message.
I remember talking about the “luminary effect” with author and consultant Boris Beizer years ago. This is where people hesitate to challenge an expert, especially a recognized luminary in their field, because of their perceived authority on a topic. But in fact, all of the experts I’ve mentioned love for you to give them your well-reasoned challenges to any of their ideas. Granted, the more they love a debate, the more exhausting and intimidating it can be to engage with them. There are smart, after all, and you need to do your homework so you can competently defend your ideas – that’s not asking too much, right? In fact, one of the best ways to get their respect is to challenge them with a good argument. I just hope that a few of my ideas here will survive their scrutiny.
In this post I’m talking about some controversial people and some controversial topics. Where I’ve stayed neutral here, I’ve done so very deliberately, and though I have some further opinions unrelated to the topics I’m discussing, I’m not going to muddy this post with them.
My team is committed to Test-Driven Development. Therefore, I was struck with remorse recently when I found myself writing some bash code without having any automated unit tests. In this post, I’ll show how we made it right.
Context: this is a small utility written in bash, but it will be used for a fairly important task that needs to work. The task was to parse six-character record locators out of a text file and cancel the associated flight reservations in our test system after the tests had completed. Aside: I was also pair programming at the time, but I take all the blame for our bad choices.
We jumped in doing manual unit testing, and fairly quickly produced this script, cancelpnrs.bash:
#!/usr/bin/env bash
for recordLocator in $(egrep '\|[A-Z]{6}\s*$'|cut -d '|' -f 2)
do
recordLocator=$(echo -n $recordLocator|tr -d '\r')
echo Canceling $recordLocator
curl "http://testdataservice/cancelPnr?recordLocator=$recordLocator"
echo
done
The testing cycles at the command line started with feeding a sample data file to egrep. We tweaked the regular expression until it was finding what it needed and filtering out the rest. Then we added the call to cut to output the record locator from each line, and then put it in a for loop. I like working with bash code because it’s so easy to build and test code incrementally like this.
After feeling remorse for shirking the ways of TDD, I remembered having some halting successes in the past with writing unit tests for bash code. We installed bats, the Bash Automated Testing System, then wrote a couple of characterization tests as penance:
#!/usr/bin/env bats
# Requires that you run from the same directory as cancelpnr.bash
load 'test/libs/bats-support/load'
load 'test/libs/bats-assert/load'
scriptToTest=./cancelpnrs.bash
@test "Empty input results in empty output" {
run source "$scriptToTest" </dev/null
assert_equal "$status" 0
assert_output ""
}
@test "PNRs are canceled" {
function curl() {
echo "Successfully canceled: (record locator here)"
}
export -f curl
run source "$scriptToTest" <<EOF
Thu Apr 02 14:23:45 CDT 2020
Checkin2Bags_Intl|LZYHNA
Checkin2Bags_TicketNum|SVUWND
EOF
assert_equal "$status" 0
assert_output --partial "Canceling LZYHNA"
assert_output --partial "Canceling SVUWND"
}
We were pretty pleased with the result. Of course, the test is a good deal more code than the code under test, which is typical of our Java code as well. We installed the optional bats-support and bats-assert libraries so we could have some nice xUnit-style assertions. A few other things to note here–when we’re invoking the code under test using “source“, it runs all of the code in the script. This is something we’ll improve upon shortly. We needed to stub out the call to curl because we don’t want any unit test to hit the network. This was easy to do by creating a function in bash. The sample input in the second test gives anyone reading the test a sense for what the input data looks like.
Looking at the code we had, we saw some opportunity for refactoring to make the code easier to understand and maintain. First we needed to make the code more testable. We knew we wanted to extract some of the code into functions and test those functions directly. We started by moving all the cancelpnrs.bash code into one function, and added one line of code to call that function. The tests still passed without modification. Then we added some logic to detect whether the script is being invoked directly or sourced into another script, and it only calls the main function when invoked directly. So when sourced by the test, the code does nothing but defines functions, but it still works the same as before when invoked on the command line. We changed the existing tests to call a function rather than just expecting all of the code to run when we source the code under test. This transformation was typical of any kind of script code that you would want to unit test.
At this point, following a proper TDD process felt very similar to the development process in any other language. We added a test to call a function we wanted to extract, and fixed bugs in the test code until it failed because the function didn’t yet exist. Then we refactored the code under test to get back to “green” in all the tests. Here is the current unit test code with two additional tests:
#!/usr/bin/env bats
# Requires that you run from the same directory as cancelpnrs.bash
load 'test/libs/bats-support/load'
load 'test/libs/bats-assert/load'
scriptToTest=./cancelpnrs.bash
carriageReturn=$(echo -en '\r')
setup() {
source "$scriptToTest"
}
@test "Empty input results in empty output" {
run doCancel </dev/null
assert_equal "$status" 0
assert_output ""
}
@test "PNRs are canceled" {
function curl() {
echo "Successfully canceled: (record locator here)"
}
export -f curl
run doCancel <<EOF
Thu Apr 02 14:23:45 CDT 2020
Checkin2Bags_Intl_RT|LZYHNA
Checkin2Bags_TicketNum_Intl_RT|SVUWND
EOF
assert_equal "$status" 0
assert_output --partial "Canceling LZYHNA"
assert_output --partial "Canceling SVUWND"
}
@test "filterCarriageReturn can filter" {
doTest() {
echo -n "line of text$carriageReturn" | filterCarriageReturn
}
run doTest
assert_output "line of text"
}
@test "identifyRecordLocatorsFromStdin can find record locators" {
doTest() {
echo -n "testName|XXXXXX$carriageReturn" | identifyRecordLocatorsFromStdin
}
run doTest
assert_output $(echo -en "XXXXXX\r\n")
}
You’ll see some code that deals with the line ending characters “\r” (carriage return) and “\n” (newline). Our development platform was Mac OS, but we also ran the tests on Windows because the cancelpnrs.bash script also needs to work in a bash shell on Windows. The script ran fine under git-bash on Windows, but it took some tweaking to get the tests to work on both platforms. There is surely a better solution to make the code more portable.
We installed bats from source and committed it to our source repository, and followed the instructions to install bats-support and bats-assert as git submodules. We’re not really familiar with submodules and not entirely happy with having to do a separate installation of the submodules on every system we clone our repository to (we have to run “git submodule init” and “git submodule update” after cloning, or else remember to add the option “–recurse-submodules” to the clone command).
Running the tests takes a fraction of a second. It looks like this:
$ ./bats test-cancelpnrs.bats
✓ Empty input results in empty output
✓ PNRs are canceled
✓ filterCarriageReturn can filter
✓ identifyRecordLocatorsFromStdin can find record locators
4 tests, 0 failures
Here is the current refactored version of cancelpnrs.bash:
#!/usr/bin/env bash
cancelEndpoint='http://testdataservice/cancelPnr'
doCancel() {
for recordLocator in $(identifyRecordLocatorsFromStdin)
do
recordLocator=$(echo -n $recordLocator | filterCarriageReturn)
echo Canceling $recordLocator
curl -s --data "recordLocator=$recordLocator" "$cancelEndpoint"
echo
done
}
identifyRecordLocatorsFromStdin() {
egrep '\|[A-Z]{6}\s*$' | cut -d '|' -f 2
}
filterCarriageReturn() {
tr -d '\r'
}
if [ "${BASH_SOURCE[0]}" -ef "$0" ]
then
doCancel
fi
There are two lines of code not covered by unit tests. Because the one test that hits the loop body in the doCancel stubs out curl, the actual curl call is not tested. Also, the doCancel call near the bottom is never tested by the unit tests. We ran manual system tests with live data as a final validation, and don’t see a need at this point to automate those tests.
It’s been too long. Hello again. I’m still working on the next installment of Jerry’s Story. I’m going to restart it once again, and some day I’m going to get it right. Meanwhile, I dug up a list of my articles, presentations, and other content from 1995 – 2009, both self-published and otherwise, and found working URLs for them (I appreciate archive.org so much!). I’m putting them here mostly for future reference, but if you do delve in, please let me know if any of the links still don’t work.
One bonus if you scroll all the way down–my very first feature article, from 1987.
There are also some posts on myTejas Software Consulting Newsletterblog that I may republish here. Many of the articles linked below have outdated contact information. I’d love to hear from the modern you either in a comment on this post or via Twitter.
So with no further ado, in reverse order of musty outdatedness, here is my long list of nostalgia.
March 2008 Gray Matters podcast (mp3) Jerry McAllister interviewed me for this podcast, where we talked about the testingfaqs.org Boneyard and the strength of the worldwide test tools market.
Trip Report from a USENIX Moocher Dallas/Fort Worth Unix User’s Group, July 2003, reprinted in the Tejas Software Consulting Newsletter, August/September 2003, v3 #4
Developing Your Professional Network The Career Development column in the January/February 2001 issue of Software Testing and Quality Engineering magazine. References from the article are available on the STQE web site.
Software Defect Isolation Co-authored with Prathibha Tammana. Presented at the High-Performance Computing Users Group, March 1998, and InterWorks, April 1998.
Tester’s Toolbox column: The Shell Game Software QA magazine, pp. 27-29, Vol. 3 No. 4, 1996 Using Unix shell scripts to automate testing. Basic information about the available shells on Unix and other operating systems.
Tester’s Toolbox column: Toward A Standard Test Harness Software QA magazine, pp. 26-27, Vol. 3 No. 2, 1996 The TET test harness and where it fits into the picture.
Tester’s Toolbox column: Testing Interactive Programs Software QA magazine, pp. 29-31, Vol. 3 No. 1, 1996 A concrete example of using expect to automate a test of a stubborn interactive program.
Tester’s Toolbox column: Using Perl Scripts Software QA magazine, pp. 12-14, Vol. 2. No. 3, 1995 The advantages of using the perl programming language in a test environment, help in deciding whether to use perl and which version to use.
Apple Kaleidoscope, Compute! Magazine, pp. 111-112, issue 91, Vol. 9, No. 12, December 1987. I was interviewed about this article for “The Software Update” episode of the Codebreaker podcast, released December 2, 2015.
As I sit here listening to Christmas music, I’m giving myself the gift of extra time to write. I want to respond to something Paul Maxwell-Walters recently tweeted:
If there is such a thing as a Tester’s Mid-Life Crisis, I think I may be in the middle of it….
Paul cited the director of a counseling center who said mid-life crises are likely to happen between age 37 through the 50s. Paul, approaching his 40s, worries that his crisis is here. As I see my 50s getting large on the horizon, I don’t know if my crisis has past, is still coming, or will never come. I was actually around Paul’s age when my consulting business dried up and I ended my 16-year run in software testing. Four years later, though, I went back to my comfort zone, and had four consecutive short stints in various testing jobs.
That last testing job morphed into a development job. I’m very happy with my current employer for encouraging that path to unfold. Over the years, I have fervently resisted several opportunities to move into development, some of them very early in my career. I had latched onto my identity as a tester and staunch defender of the customer, and I wouldn’t let it go.
Paul wrote:
I have also come across people around my age and older who are greatly dissatisfied or apathetic with testing. They feel that they aren’t getting anywhere in their careers or are tired of the constant learning to stay relevant. They feel that they are being poorly treated or paid much less than their developer colleagues even though they all work in the same teams. They hate the low status of testing compared to other areas of software development. They regret not choosing other employers or doing something else earlier.
That’s surely the story of any tester’s career. Low status, low pay, slow growth. I embraced it, because I loved the work and loved what it stood for. The dissatisfaction seems to be more common now than it used to be, though. My advice, which you will know if you’ve been reading things on my blog like “The black box tester role may be fading away“, is: get out! Don’t transition to doing test automation. Become a developer, or a site reliability engineer, or a product owner, or an agile coach, or anything else that has more of a future. I think being a testing specialist is going to continue to get more depressing as the number of available testing jobs slowly continues to dwindle.
Because I’m writing this on Christmas Eve, I want to put an It’s a Wonderful Life spin on it. What if my testing career had never been born? In fact, what if the test specialist role had never been born?
Allow me to be your Angel 2nd Class and take you back to a time when developers talked about how to do testing. Literature about testing was directed toward developers. What if no one had worried about adding a role that had critical distance from the development process? What if developers had been willing to continue being generalists rather than delegating the study of testing practices to specialists, while shoving unit testing into a no-man’s land no one wanted to visit?
And what if I could have gotten over the absolute delight I got from destroying things and started creating things instead? I’m sure I’d be richer now. I’d have better design skills now. But alas, I’m not actually an Angel 2nd Class, and more to the point, I haven’t dug up enough historical context to really play out this thought experiment. But I’ll try to make a few observations. Within the larger community of developers, I might not have been able to carve out a space to start a successful independent consulting practice, which I dearly loved doing as a tester. Maybe I wouldn’t have developed my appreciation for software quality that I have now. Maybe I wouldn’t have adopted Extreme Programming concepts so readily as I have, which has now put me in a very good position process-wise, even if I’m having to catch up my enterprise design and architecture skills.
How about not having any testers in the first place? Maybe the lack of critical distance would have actually caused major problems. Maybe the lack of a quality watchdog would have allowed more managers to actually execute those bad decisions. And maybe those managers would have been driven out of software management. Would the lack of a safety net have actually improved the state of software management by natural selection, and even allowed some companies with inept executives to die a necessary death? I think I’m hoping for too much here, and perhaps being too brutal on Christmas Eve.
It has been a wonderful career. It could have been a different career, but I’m just glad that it has taken me to where I am now. Paul, I wish you a successful outcome from your mid-career crisis. I realize that my advice to get out is much easier said than done.
Another one of my mentors is gone – I got the news that Boris Beizer passed away on October 7, 2018. I’d like to pause to share some of my recollections of Boris. If you knew him, I would love to hear your stories, too.
I think my first introduction to him was reading his book Software Testing Techniques. It was published before the software testing specialist role was common. I was working as a software test engineer, and I was a bit confused by the book’s point of view. I discovered that Boris and most of the other authors who wrote about software testing at the time were participating in the comp.software.testing Usenet newsgroup. This was likely in 1994, give or take a year. I was amazed that I could interact with the people who “wrote the book” on software testing. So I joined in, and I learned a lot more than I would have just from their books. Somewhere along the way, Boris explained that Software Testing Techniques was written for programmers, and suddenly it made a lot more sense to me. When I wrote the frequently asked questions list for the newsgroup, I used quite a bit of material from Boris to flesh it out.
In 1995, I set up the swtest-discuss email list that Mark Wiley and I conceived to discuss how to test operating systems with a few colleagues we knew. The list grew to 500 subscribers and the topic area greatly expanded. Some people liked how we could enforce a better signal to noise ratio than what we had on comp.software.testing. Boris participated on the list. But some people felt that his tone was too abrasive. I’ve forgotten the details of the social dynamics that were in play so long ago. Some people moved on to other forums where Boris wasn’t invited. I realize I can’t make everyone happy. And Boris clearly didn’t care to.
My participation on Usenet got the attention of Dr. Edward Miller, the conference chair for the Quality Week conference. He invited me to join the conference’s advisory board that chose the papers that would be included. I was flabbergasted. I was still practically a kid. But Dr. Miller was certain he wanted me on the board. So I accepted. I joined a distinguished group of industry experts and academics, including Boris Beizer, who was a prominent industry consultant and also still acted like an academic, having gotten one of the first ever PhD’s in computer science.
I traveled to the Quality Week conference in San Francisco, which was in the Sheraton Palace. I remember going to the dinner that the advisory board was invited to during the conference each year as a thank-you for our efforts. I wasn’t sure how I was going to get to the restaurant as I stood on the curb in front of the hotel with Boris and other board members, many of them smartly dressed, and me in my business casual. Then Boris hailed a limo. What? I didn’t know then that you could hail a limo, but that’s how several of us got to the restaurant. Edward and Boris and the rest accepted me as one their own, despite my inexperience and casual mode of dress.
Some of the specific things I remember from Boris include the Pesticide Paradox. which taught us that test suites lose their effectiveness over time. His software bug taxonomy inspired many discussions, and I even helped him research the origin of the word “bug.” He taught me that if I can model any aspect of a program using a graph, I can use that graph to guide my testing. And not long ago, at a talk I was giving, someone in the audience reminded me of the fabulous poem “The Running of Three-Oh-Three.” Boris published it at the very beginning of Software Testing Techniques, “with apologies to Robert W. Service.” It remains the best poem about software testing that I’ve seen. I’ve only now bothered to figure out the link to Robert Service; it seems that Boris’ inspiration was Service’s poem “The Cremation of Sam McGee,” published in 1907.
Boris must have been in high demand. He told me at one point that he sold his services in 1-week blocks for US$20,000. Any shorter time than that wasn’t worth his attention. He told me later that he had enough “f-you money” to be very selective about which clients he took. He is credited with changing the industry in ways I don’t even understand, because the transformation was well underway when I joined the scene. With his brash nature, he made enemies along the way. But I didn’t like to choose sides. I have learned both from Boris and many of the people who steered clear of him.
I am especially proud of the inscription that Boris wrote in my copy of his book Black Box Testing:
However, after I read the book, I had to report to him that I really didn’t like it. He explained that the publisher had assigned him an inexperienced editor who made a wreck of the book. I sure learned a lesson about dealing with publishers.
I found out at some point that Boris had written two fiction books under the pseudonym Ethan I. Shedley. They were both out of print, but I found a used copy of The Medusa Conspiracy. I started reading the book but didn’t finish it. I probably don’t have the generational context to be able to appreciate it.
Ever since Boris retired some time ago, I’ve wondered if we would ever hear about him again. Last February, I felt an urge to check on him. I no longer had a working email address for him (he seemed to change his email account regularly), but his phone number was easy to find in his Usenet signature. Dialing a phone is a quaint thing nowadays, but I was determined. Sure enough, someone picked up the phone, and when she asked who was calling, I hastily had to summarize who Boris was to me. She summoned him to the phone and we had a nice talk. I mentioned that I’m writing a biography, and as soon as it came out of my mouth, I felt that awkward sensation that I’ve felt a few times before, that I was talking to someone who may merit a biography of their own, but yet they hadn’t made the cut. Boris mentioned his last book, “Software Quality Reflections.” I still didn’t have a copy (it may have been an e-book), and I think the only way to get one is to get it straight from him. I sent him an email to his new email address to request it as he asked me to do, but I never got an answer.
We’ve come full circle, with Boris ushering in the age of the testing specialist, and now as he makes his exit, testing efforts are shifting right back to the developers he originally addressed. I think his goals are well-stated in the dedication that he wrote in Software Testing Techniques. I’ll let him have the last word–
Dedicated to several unfortunate, very bad software projects for which I was privileged to act as a consultant (albeit briefly). They provided lessons on the difficulties this book is intended to circumvent and led to the realization that this book is needed. Their failure could have been averted—requiescat in pace.
I have given myself a challenge that I’m enjoying right now – immersing myself more deeply in the Wikipedia community. The culture that has developed around the people who add to and edit Wikipedia bears some resemblance to the culture that we see with users of services like Facebook, Twitter, and Slack, but it’s much more complex. I aim with this article to convey a sense of the richness an online community can develop, and the frustration that an outsider can feel, much like a physical community. Maybe I’ll convince you to be a Wikipedia editor too.
First I’ll mention another online community that encourages its members to make contributions that everyone can benefit from. I made an attempt to become a productive contributor on Stack Overflow, a web site for asking and answering questions about computer programming.
Stack Overflow uses a reputation system, where various contributions you make will increase your reputation score. Additional features becomes available when your reputation grows. I thought that building a good reputation on Stack Overflow could be something I could add to my resume. I answered a few questions, and was able to especially build reputation when I answered questions that no one else had answered. But I had trouble finding questions that hadn’t already been thoroughly answered within minutes of being posted. And I was chastised a few times for not strictly following the rules when I posted answers, which was disheartening when I had put effort into answering the question. I understood that a community should have rules, but I lost interest before I really learned enough of the rules to be productive on the platform. I went back to only reading the content on the site, which, like many programmers, I do often.
I created my Wikipedia account way back in 2003. According to the contribution log, which shows almost every detail from the beginning, this must have been because I wanted to add a new article about load testing. I had made some edits to other pages without an account, but I needed to have an account to create a new article. That article is still there today, and I’m happy to see that after hundreds of edits, some of my original phrasing is still there.
Wikipedia newbies would be wise to hold off on creating new articles, however. I have added a total of seven articles. Of those, two have been converted to redirects to other articles with a broader scope, and one was deleted. I created an article about Brian Marick in 2007. Amazingly, it lasted until 2016 before someone claimed that he was not notable and eventually got it deleted. Recently, someone created an article about Janet Gregory, and within 24 hours, someone started a proposal to delete it. It was deleted 8 days later. For many topics, perhaps especially for articles about people, it’s quite difficult to prove that they meet Wikipedia’s notability guidelines. It must be sad when someone sees that they’ve been declared non-notable. I will probably not be adding new articles any time soon.
My long tenure on Wikipedia has caused others to assume that I’m well-versed in the rules of the road, but with my off-and-on interest in contributing, I’m just beginning to absorb the elaborate set of rules and conventions that editors are subjected to. In response to an edit I made to the article on Jerry Weinberg that was not up to snuff, one editor told me “Surely you know by now that ‘he told me so himself’ is not considered to be adequate sourcing for anything here…” Unlike a traditional encyclopedia with articles written by trusted experts, everything on Wikipedia is expected to be backed up by published independent sources.
I find it rewarding to make edits that improve the content on Wikipedia. Doing the research to justify the edits helps me to build my knowledge. I can refer to the article later when I want to refresh my knowledge of a subject, and so can anyone else. But the benefits are greatly reduced if the changes aren’t accepted by the community. In the case of my edits on the Jerry Weinberg article, I swallowed my ego and asked how I could change my approach. This led to a healthy discussion, and I was successful when I used a different approach. It’s often not easy to engage in this sort of discussion when I’m still smarting from having someone erase my work.
I recently decided to get more involved with Wikipedia because of efforts by Noah Sussman and Alan Page. Noah put a lot of effort into improving the Wikipedia article on software testing. Then Walter Görlitz, another volunteer Wikipedia editor, removed a big edit that Noah made to the article, an action that Wikipedia calls a “revert.” A heated discussion ensued, and little progress was made on overhauling the article like Noah wanted to. Alan issued a call for help to figure out how to effectively improve the article (A call to action: let’s fix the Wikipedia page on software testing), and I and several others joined a discussion on a chat group outside of Wikipedia to strategize.
I decided to take an agile approach; make small changes and watch to see how the broader Wikipedia community responds. A few of us have also branched out and looked at the many other articles related to software testing and found that as a group, they aren’t very well coordinated or well written. We’ve made numerous small changes to these articles now with a great deal of success, though we haven’t made substantial progress on the original goal of completely revamping the software testing article. I’m starting to make somewhat larger edits now and I hope others do too.
Instead of wondering how Walter would react to my edits, I decided to engage with him directly, so I invited him to open a dialogue. I’ve seen that he puts a great deal of effort into improving and reverting vandalism on a large number of Wikipedia articles. Walter tells me that the small edits I’ve made successfully has improved my reputation with him, and he is now less likely to revert the changes I make. Unlike Stack Overflow where reputation is tracked in one place, on Wikipedia your reputation is earned individually with each editor.
I’ve found that the best way to get consensus on Wikipedia is to use the “talk page” feature on the site itself, so everyone who is following the changes on the page have an opportunity to respond. In fact, the Wikipedia community prefers for discussions like this to take place on Wikipedia, otherwise some editors may get suspicious that one person is trying to recruit a cabal of “meatpuppets” to artificially amplify their influence. Our chat group is very loosely coordinated, and the edits we make are all an individual decision, so I’m not worried that we’ll raise this suspicion, especially now that we’re using the talk pages more.
The response to an edit can vary significantly based on who is watching for changes on the article. Many areas don’t happen to have anyone watching closely, so whether the edit is useful or not, it may stay around for years. If someone feels strongly about maintaining the quality of a particular article, you’ll be held to a higher level of scrutiny. I’ve also found that newly added information may be held to a higher standard than what is already in an article. Once I added some information to an article without including a citation to back it up, and the information was removed, despite the fact that most of the information already in the article was also not referenced. It was just easier for the editor who did it to revert a recent edit than to address the broader problem. You can avoid this if you scrutinize your own contributions very carefully.
If you’re afraid you’ll have difficulty getting edits on Wikipedia to stick, take this to heart, from the instructions to “Be Bold” with your edits:
Think about it this way: if you don’t find one of your edits being reverted now and then, perhaps you’re not being bold enough.
Now if you’ll excuse me, that load testing article has become a bit of a mess.
My thanks to Simon Morley and Walter Görlitz for helping me improve this article.
“The project structures they describe seem to be on the leading edge of the future of the software testing role. In my limited view of the software industry, I don’t see many companies that are anywhere near Microsoft in their evolution.”
They don’t think they’re really on the leading edge. I suspected that they would protest, because it’s human nature – everything is relative. If the organization I’m working with has a long way to go to achieve something that theirs is already doing, and I feel that theirs is a good enough exemplar, I don’t need to seek out something that’s even better. But from their point of view, they want to improve with the help of others who have gone even further, so their sights are set on others who have gone beyond. I’ve seen many software organizations recently that are so far behind in their evolution that it’s easy for me to point to Microsoft (which granted, I’ve also found many reasons to malign) and say that they’re far ahead of the pack.
These organizations that are behind the curve are generally surviving, and their companies are usually still making a profit. I saw this many times in my consulting. They are more or less successful and often don’t feel a strong need to make significant changes. This is where the traditional approach to software testing will hang on for a long time and keep the black-box testing role alive. At least for me, though, these are often not desirable places to work, and they may eventually find that it’s difficult to hire talented testers. Note that I’m more concerned about some of the antiquated traditional practices like scripted manual testing than I am with the black-box tester role itself, for now.
By the way, I’m amused that Alan introduced me as “our good friend,” but then didn’t seem to know whether that was okay to say. I’ve found that calling someone a friend often makes it so. I owe you guys a hug.
Is the traditional software tester role fading away? A recent blog post from Cem Kaner helped to reinforce this idea for me. You’ll find his comments inside this long post, in the “2. Oracles are heuristic” section, under the heading “A supplementary video”: Updating to BBST 4.0: What Should Change. Incidentally, the topic of the whole post is updating a course on software testing, which I think I attended a very early variation of around the turn of the millennium.
Cem’s parenthetical notes on tester careers are interesting. He suggests that traditional black-box testing (whether exploratory or scripted) will give way to piecework, where a tester will be paid by the number of completed tests. I’ve seen this model already underway in outsourcing companies like Rainforest QA and its competitors, where the manually executed test step is the basic unit that you’re paying for. It’s much easier to see this happening for scripted tests than for exploratory tests, and I argued against using this kind of service when an executive asked me to consider using it, because I see little value in scripted manual tests.
You can imagine that this kind of piece work will not pay testers very well. Cem noted that he already sees a significant pay differential between black box testers who don’t do any programming and those who have jobs that require some kind of programming. He suggests a few skills like programming that testers could add to become more marketable. I have a few items of diversification I can point to, including programming and automation skills, though I don’t often focus on automation. I’m familiar with testing web apps and mobile apps (and even mobile web apps :). My experience with embedded systems often gets the attention of recruiters.
I’ve added a few items to my resume this year that I’m happy about. I took a training course and became a Certified Scrum Master so I could understand the leadership aspect of agile processes better. Perhaps the most promising, given the current job market for security, is the Certified Ethical Hacker course I took and passed the exam for. I know that passing a certification exam doesn’t really prove anything, but these particular certifications do seems to carry some weight that might help my career. Both are subject areas I already have experience in, and I was happy to round out my knowledge in the classes.
I’ve been following Microsoft employees Alan Page and Brent Jensen on their AB Testing podcast. They both have had fairly traditional testing roles, but now are in roles that seem to be much more future-proof. Brent is now a data scientist, specifically, Principal Data Scientist Manager. Alan, Principal Software Engineer, describes himself as a helper, doing the odd (but challenging) tasks that don’t easily fit the developer roles on his team, which is something that appeals to me. Both are still involved in the testing process. The project structures they describe seem to be on the leading edge of the future of the software testing role.
In my limited view of the software industry, I don’t see many companies that are anywhere near Microsoft in their evolution. If the traditional black-box tester role is fading, I think it’s going to happen very slowly. I think it will require a very broad view of the industry to track a slow evolution like this, and I’m curious if you’ve heard from anyone who is in a good position to see it.
I have been working on honing my security testing skills. I asked Don Ankney‘s advice on how to do this, and one of his suggestions was to participate in bug bounty programs. Many companies encourage security researchers to report security vulnerabilities to them, and in some cases, they offer monetary rewards to the first person who reports each one.
My first bug bounty report for Instagram, which wasn’t accepted, was discussed here: “Username Enumeration – Ho, Hum!” This time, though, I was more successful. I found that none of Instagram’s cookies on its web interface had the “secure” flag set, including the session cookie that identifies a logged-in user. Like username enumeration, the secure flag on the cookies is another “ho, hum” thing often excluded from bug bounty programs. But the Facebook Bug Bounty Program (which also covers Instagram) doesn’t mention such an exclusion, so I decided to report the vulnerability.
I spent some time crafting an attack scenario. I found that the attack didn’t work if I used “instagram.com” instead of “www.instagram.com.” I found that if the insecure page http://instagram.com was in the browser cache, the browser used the cached page and then there was no vulnerability. And for reasons I haven’t figured out, I was not able to complete the attack successfully if the victim was using Firefox. I was able to prove that hijacking an Instagram session was a simple matter of setting just the captured sessionid cookie. This is the bug report I sent:
Description and Impact
The secure flag is not set on any of Instagram’s cookies, including sessionid. When a user with an active session types “www.instagram.com” in their browser to go to the site, they will first hit the insecure site and transmit all of their cookies in the clear. An attacker monitoring their network packets will be able to hijack their session easily. Assuming there is no need to send cookies in the clear at any point, this is easily fixed by setting the secure flag in the cookies.
Reproduction Instructions / Proof of Concept
I implemented a proof of concept using Safari 8.0.8 on Mac OS 10.10.5 and Chrome 49 on Windows Vista Home Basic for the victim. I haven’t been able to reproduce it yet with Firefox.
Make sure you’re not logged in to Instagram. Clear the browser cache.
Click “Log in with Facebook”, and enter valid Facebook login credentials. This logs you in to Instagram.
…an arbitrary amount of time may pass, as long as the Instagram session is still valid when continuing.
Go to a public network that someone is snooping on.
Open a tab in the same browser as before and go to http://www.instagram.com (not https). The sessionid cookie is sent in the clear and has been captured by the attacker. Even though the server returns a 301 redirect to a secure site, the cookie has already been sent in the clear.
Attacker hijacks the Instagram session by setting the sessionid cookie in their browser.
I got a reply five days later, saying “This is currently intentional behavior in our product…” I wasn’t surprised that another “ho, hum” bug was rejected, but I was surprised that they considered it a feature. So I replied, saying that I intended to publicly disclose the issue (which is standard practice after the report is closed, whether fixed or not) and I asked for further information about how the site needs this behavior in order to function, to inform my continued testing. I call this sort of response my “Just one more thing” reply, inspired by the TV character Columbo. This sort of followup is routine for professional software testers, but I don’t know how many security penetration testers put bug advocacy skills to use.
The next reply came quickly, saying that though many people had already reported this issue, they would go ahead and discuss the issue with the product team and try to fix it. And lo and behold, about three weeks later, I got notice that the issue is resolved, and I was pleasantly surprised to hear that they offered to pay me a bug bounty. The reasoning was fascinating – the site previously used http (I’m not clear how long ago) and then later switched to https. All the previous reports about this issue had been when they used http, which is silly, since in that case the secure flag would render the cookies invisible to the server. This explains their earlier pat rejection of bug reports about the secure flag, even though that response had become obsolete with the change to https.
They determined that I was the first to report the vulnerability since they switched to https, and so I qualified for the bounty. I am impressed with the amount of care that Facebook/Instagram took in handling this report. I’m eager now to dig deeper and apply more of my bug advocacy skills if necessary.