Tag Archives: testing

Agile Testing Days 2015

November in my work calendar means only one thing: Agile Testing Days 2015.

This was the 3rd time I participated in ATD, although this time as a speaker which was a really interesting experience.

As usual what is always great at ATD are the conversations and I was lucky enough to speak to Anne-Marie Charrett and find out how she has been coaching testers at her organisation and the challenges she has faced hiring new testers. The MIATP award party on Tuesday night was also brilliant and it was good to meet some new people from around the world.

One of the highlights of the conference was playing the pen game with my colleague Andrew Fraser (who has also published his conference notes on his blog, which are a far better representation than mine! check them out here) and playing some testing games on the Wednesday night!

I will write a separate blog post about the talk I presented but here are my (short) notes taken during some of the keynotes and talks I attended.

This year I also missed the tutorials day due to schedule restrictions so I’ll go straight to day 1. I hope you find these notes useful, but if you have any questions or want some clarification feel free to leave a comment!

EDIT: You can now find the slides from all presentations here.

Day 1

“Where words fail music speaks” Keynote by Huib Schoots and Alexandra Schladebeck

  • what’s common between music and testing
    • instant feedback
    • reduce complexity to practice
    • learn fundamentals (own them)
    • muscle memory
    • you don’t practice from front to back
  • teams have the same soundtracks, they are on the waves (i.e. understanding)
  • hero complex, music teams and their contexts
    • solo
    • trios/quartets
    • bands
    • orchestras
    • sessions
  • STAR heuristic
    • structure
    • tune
    • accompaniment
    • rhythm
  • know what your cadences are… (authentic and deceptive, like musical notes)
  • exploratory testing and music:
    • scripting versus improvisation
    • communication over documentation
    • standards and music
  • experiments: outcome is unsure
  • non-deterministic and non-reproducible
  • models: based on experience, culture, etc.
  • lessons:
    • learn your team’s tune
    • finish your sprints on a good cadence
    • recognise your role and context
    • practice, patterns, adapting = you’ll be a star


“Experimenting in context for Exploratory Testing” Talk by Maaret Phyajarvi

  • replacing a test case driven style with a learning tester driven style in two organisations
  • what testing gives us:
    • unit testing: spec, feedback, regression, granularity, testing as artifact creation
    • exploratory testing: guidance, understanding, models, serendipity, testing as performance
  • data intensive application
    • things that couldn’t be changed at certain organisations (the “givens”)
      • waterfall process
      • contractual distance between acceptance testers and subcontractor
      • test case metric based reporting
      • manage, don’t test
      • business end users as testers
    • things that have changed (the “experiments”)
      • acceptance tester degree of freedom
      • test cases from step by step scripts to test data with process outline for notes
      • making change requests acceptable
      • reporting ~20% of testing to 3rd party
      • unofficial tips sharing sessions with the subcontractor
  • Function intensive application:
    • “Givens”
      • roadmapping creating disconnect to current priorities
      • tendency for remote work
      • developers doing majority of testing
      • requirements / specifications format as UI spec
    • “Experiments”
      • no test cases or wasteful documentation
      • tester with developer tools
      • removing “acceptance testing” by moving testing to the team
      • continuous delivery (without test automation)
      • holding space for testing to happen
      • true teamwork with mob programming

    • past
    • results
    • obstacles
    • outlook
    • feelings
  • vision, current charter, other charters, details (bug reports)
  • charters to give ideas of exploration


“A happy marriage between context-driven and agile” Talk by Ilari Henrik

  • Automate everything?
  • checking vs. testing
  • checking the algorithmicly catchable stuff
  • TDD is not testing
  • automated check:
    • passed – ok or missing a bug
    • failed – there is a problem or false alarm
  • “Embedded testers”
  • Context-driven and agile manifesto
  • context driven testers will “click” really well in agile teams
  • ilari.com/agile PTE Agile Testing Manifesto
  • how we do it:
    • get involved early
    • bridge between developers and testers
    • pair on tasks
    • educate the team about testing
    • technical awareness “I comment my code – but I still don’t know what I did or why”
    • domain knowledge
    • willingness to learn
    • when crafstpeople meet other craftspeople, that’s when the magic happens


“Testers are dying” Keynote by Karen Greaves and Sam Laing

  • There’s more demand than there’s capacity – testers aren’t dying
  • 6 steps:
    • awareness
    • punishment (different tariffs, rates going up)
    • personal responsibility
    • remove barrier
    • visible impact
    • start a movement
  • trends:
    • pressure at the end of the sprint
    • often asked to release without testing
    • testing is always behind development
    • blame around bugs
    • automation is at least a sprint behind
  • personal responsibility by different boards – different columns like “show me”, “review (in pairs)”, different colour coding


“The human refactor experiment” Keynote by Bryan Beecham

  • horizon of predictability: we can only see so far into the future
  • moment of maximum ignorance
  • get past mental limits – other people will start buying in
  • 23 1/2 hour challenge
  • refactoring:
    • exercise
    • diet
    • continuous improvement
  • make the world a kanban board


Day 2

“If I can do it so can you” Keynote by Dr. Sue Black

Dr. Black told the audience the story of her life right from growing up in the suburbs of London to forming a campaign that helped save Bletchley Park, the home of the World War II code breakers. Really inspirational.


“Shift left and shift right – the testing swing” Talk by Laurent Py

  • journey from waterfall to agile
  • breaking silos
    • speed of feedback loop
  • put value before correctness
    • why? is it worth doing?
    • how to ensure quality of deliverables?
  • testing matrix
    • build / production
    • what should we buid and why? is it really worth it?
    • how to automate? is it really reliable and does it scale?
  • good for challenging business assumptions


“There is no secret sauce” Keynote by Ben Williams and Tom Roden

  • investing in impact model
  • are we investing in the right things?
  • what aren’t we going to do now?
  • investing in impact – a leaner approach to investing in software
    1. decide to invest – explicitly acknowledge hypothesis (between writing software and making money), consider this investment in the context of others
      • diversification: people’s skills, technical component
    2. establish performance boundaries
      • ranged planning
      • pro actively manage risk
    3. measure income
    4. measure impacts
    5. decentralised decision making
      • steer by exception
    6. learn


“Test beyond quality, beyond software” Keynote by Mike Sutton

  • Steve Blank’s book “Startup Owners Manual Step: The Step-By-Step Guide for Building a Great Company
  • agile teams are a place to cultivate behaviours and cultures for a new reality
  • it’s time for agile teams and testing discipline to go beyond software, beyond testing and into organisations
  • ask questions, go where they lead
    • “Why does it take so long?”
    • “What if?”


Day 3

Unfortunately, after giving my talk I didn’t take any notes from the two keynotes I attended. The first one was by Tom Bradford “Nowhere and back again: a software engineer’s tale” in which he described his career in software (development) and how he tried to quit before only to go back and hope to make things better. The second one, which was also the final keynote of the conference was from Olaf Lewitz “Integral Quality” in which he talked about quality across all parts of the system and the organisation and what we could do to change that.


Agile Testing Days 2013 – Day 3 Talks Notes

In the last day, after being in Germany for just over 4 days, I decided to still attend the morning talks and take a look at the sketching workshop (which I left after the break: I was tired, and I’m really poor at drawing, which meant that I couldn’t even concentrate on what we were asked to do, so I thought I’d give it another go another time). Here are the key points in the morning talks I attended.

“Natural born testers. Are you one? If not, then become one!” by Graham Thomas

  • A natural born tester is someone who tests by default. Whatever. Not destructively, or maliciously, just out of habit, or compulsion, a what if?;
  • who is a natural born tester in this picture?
    • 010
    • Hopefully you guessed (3)
  • why lemmings?
    • skills – multi-tasking, parallel processing, problem solving, time management, goal oriented, fun
  • why play railroad tycoon?
    • all about planning, management, different views by context
    • monitoring, measuring and predicting
    • controlling
    • adapting to change
    • reacting to change
    • fun
  • angry birds:
    • teaches you to explorer your content
    • simple solution is not always optimal
    • different techniques
    • combine techniques
    • plan
    • think in the abstract
  • playing angry (test) birds – hit different parts of the code
  • learn through play – raspberry pi, penguin puzzle

“Don’t you trust me?” by Seb Rose

  • Go through the behaviours with the business, everyone involved, stakeholders, look for the knowns and the unknowns;
  • our systems can be described as behaviours of our system;
  • Cucumber is good because it will bring everyone together to specify software – developers, testers, BAs, product owners;
  • it also helps you give live documentation which is why it has some advantage over other tools;
  • what is the problem with this collaboration in BDD?
    • some places aren’t quite as agile as they think they are
    • talking to each other – BDD actually helps with this because you need to speak to each other
  • look at different components, don’t just test drive them;
  • regaining trust
  • too many organisations are; agile in the way that are not what we would like to think about it – they are still too structured;
  • Ron Jeffries – No Estimates;
  • testing pyramid (uni/integration/end-to-end/exploratory&manual);
  • ice-cream collapse pattern
    • 006
  • don’t treat acceptance tests as system tests – both are different and have a different audience;
  • be careful what you test with BDD – it’s expensive, certain things you can go directly to the method and test it there

Agile Testing Days 2013 – Day 1 Talks Notes

In the first (well technically second if you count the tutorial day) day of the conference I attended the 2 morning talks and in the afternoon I floated between consensus talks and the various workshops and vendor boots. Here are the key points talked about in those talks.

“How to avoid the testing swiss cheese syndrome” by Marc Rambert

  • We’re not born testers – we become testers;
  • 2% change in a full release (development effort), how much testing effort is required?:
    • 10 test cases related to new features and bugs fixed in this release
    • 90 regression test cases
    • solution: there is no relation – it’s too difficult to align testing and coding
  • speed and continuous delivery make it impossible to test everything after each change;
    • change request implemented and tested (build 0)
    • functional regression set #1 (build 1)
    • bug because of last minute effect (build 2)
    • go live!
  • strategies to focus testing where it adds value (requirements, risks, experience, collaboration)
  • a new opportunity to improve testing in a black box:
    • learning system: learn your tests as usual but capture footprints (link code and test)
    • detection: application changes
    • smart engine
    • you can also add tests that have been run before
  • test scoring to prioritise test execution
  • avoid the testing swiss cheese syndrome: find a way to make your application speak

“Be a real team member” by Tony Bruce

  • What makes a good team member:
    • engage, and use that to build a relationship, interest and motivate people (4 keyword framework)
    • motivate: provide someone with a reason for doing something
  • Models (engage)
    • Belbin team model – action oriented roles, people oriented roles, thought oriented roles
    • plant is someone that comes up with the idea
    • resource investigator, co-ordinator, shaper, monitor evaluator, team worker, implementer, completer finisher, specialist (they key is balance)
  • as you learn more your role will change too
  • Margerison-McCann model
  • day to day:
    • positive action over positive thinking – do it rather than mention it / think about it
    • ask the questions!
    • feedback: express what you do want, rather than what you don’t want!
    • reciprocation: essentially states that if someone gives something to us, we feel obligated to repay that debt; give help; ask for help (give and take)
    • always acknowledge, never dismiss or ignore
    • don’t assume that only people with higher jobs than you have valid opinions
    • beware of the curse of knowledge
      – cognitive bias
      – can be off-putting
      – can leave people feeling dejected
      – why the should care?
    • act as a sounding board
    • appreciate any input
    • beliefs followed by behaviours
    • find people who work because they believe people over money
    • always able to offer different perspectives
    • invest time with people whose work crosses organisational boundaries
    • breaking bread – sharing your lunch – best ideas are shared over food
    • remain reliable
    • listen – don’t just sit around with your headphones on – listen and eavesdrop
    • do the i’s and cross the t’s
    • problems don’t lie in the philosophy of procedures but in practice, and practice is governed by attitude
    • before you speak think – is it true, helpful, inspiring, necessary, kind?

Consensus Talks

“Group Testing” by Christian Baumann

  • Regression testing, how to overcome its error proneness and boredom?
    • no testers in the team
    • group testing! everyone involved, tests distributed randomly, everything until finish and debrief
  • benefits: concurrency and performance issues detected, no one is testing alone, safety net before release;
  • how often do you do it and when? it got forgotten;
  • lists get too long, big tests vs. small tests;
  • regression testing was done in areas where automated tests are lacking;
  • executed regularly (frequency depends on findings);
  • not too boring or repetitive;
  • unsolved issues:
    • decreasing motivation
    • retrospective not done regularly any more
    • number of tests growing
  • officially it was meant to happy every 2 weeks but it just didn’t;

“Are we still testing the wrong stuff?” by Stephan Kamper

  • There’s more to test than what’s desired today
  • two values of software:
    • ability to tolerate and facilitate such ongoing change is the primary value of software, it has to be soft
    • build the software without too many bugs (we’re ok at this [-ish])
  • however, the 1 value people keep saying you ain’t gonna need it (yagni);
  • everyone (test, ux, ba, managers, etc.) should care about eh primary value too;
  • but, few teams do this kind of testing, future readiness not that important after all? relation to software life time?
  • we need zebricon: there’s no answer but may be a concrete answer isn’t the point;
  • how about testing future readiness?

“So I am an Agile test manager now… but what does that mean?” by Mitch (surname unknown)

  • Manager could manage test cases… but you would be a tester in that case;
  • could also manage tests… but you would be a test co-ordinator;
  • could also manage testers… but you would be a people manager;
  • strategies and guidelines: manage environment, boundaries around team, strategy and guidelines for self organised teams, impact mapping;
  • 3 amigos idea in a test manager.

Agile Testing & BDD eXchange 2013 – Notes

Last year, in November (2013), I went to the Agile Testing & BDD eXchange that was held in London and hosted by SkillsMatter.

Here are the notes and main takeaways I took/found during the day, organised by each of the 6 talks.


“How I learned to stop worrying and love flexible scope” by Gojko Adzic and Christian Hassa

  • Get organisations to benefit more from iterative delivery, allowing flexibility. You can see flexibility is everywhere apart from software, e.g. plane tickets where you can choose to pay more to be allowed to change details of the flight at a later date.
  • “Software people” tend to complain about not having enough time, even though most times it’s perceived that money is the problem where in fact time available is.
  • It’s important to change the mental model around flexibility and starting to engage with the business – this will avoid peaks in the backlog.
  • We need to change the ways to write user stories in order to avoid them becoming issues.
  • Adapt: Why success always starts with failure book recommendation.
  • Palchinsky principles:
    • first thing is to plan for variation: see out new ideas and try new things;
    • survivability: when trying something new, do it on a scale where failure is survivable;
    • selection: seek out feedback and learn from your mistakes as you go along.
  • The role of user stories is survivability, we have to get the most of them and make them survive.

Gojko then talked a bit about road maps and related back to survivability of user stories. He explained that road maps are only what we think road maps are when we talk to “software people”. For those people, roads maps are just roads, not maps. And actually they are not even roads – you go out one way and you come out the other, so they are more tunnels than they are roads. Features are delivered linearly, and for road maps to stay truthful to its meaning we need different routes, where stories are survivable experiments. User stories would then become our turn by turn navigation, instead of becoming just a destination. The sat nav analogy is quite apt here, where the more sources of feedback (GPS signal, traffic information, peer to peer information, etc.) we have the better the navigation is.

User story example:

As a sales manager
In order to monitor inventory
I want […] report

“In order to monitor inventory” is the part we have to change – there is no space for feedback here and there needs to be, otherwise we don’t know if we have made the right “turn”. Once we have a change in our story, we have a success criteria.

Impact mapping:

003 001


  • Slice your backlog for impacts
  • How much would you pay for information on whether this thing you are working on is giving you what you want? We can probably change how budgeting works.


“Whose truth is it anyway.” by Matt Wynne and Seb Rose

I was really looking forward to this talk, I had met and listened to Seb Rose only a few weeks before at Agile Testing Days 2013 in Potsdam, Germany and I had started a few months before to read Matt’s “The Cucumber book”.

Here are some of the highlights:

  • Differences between technical and business people demonstrate the value of establishing trust between everyone.
  • Iterate on a feature and show the value of it to the business – new features or changes are feedback from the business on their own.
  • Writing BDD scenarios will help you keep in the problem domain.
  • Most people that are excited by BDD and start practising it are coming from the point of view of test “automation”, which makes it a problem.
  • Declarative vs. imperative examples.
  • web_steps are dead.
  • Ubiquitous language is one of the biggest BDD selling points.
  • If we can understand the problem, the solution should take care of itself – apply problem solving skills.
  • “Whose domain is it anyway” Dan North’s blog post.
  • In order to write BDD scenarios you have to find a compromise between everyone in the team.
  • The need for detail might indicate a lack of trust.
  • Level of detail is hard to define – where are you in the pipeline: know all the details in the system or know nothing, it’s important to establish the compromise.
  • Business people do care about the behaviour of the system.
  • Easy is different than simple:
    • Understanding this, and getting there, helps you understand the project and your tests;
    • Time saved at the beginning is the most important of the project.

The testing iceberg:



“What do testers do?” by Tony Bruce

Tony is a good friend of mine, someone that I also met at my first ever conference, Agile Testing Days 2013. He blogs here, and you can find the “slides” for this presentation here.

Main points:

  • Testers don’t:
    • randomly click things;
    • check off requirements;
    • represent a safety net or a gateway;
    • if automation works “well” people might not believe you need to do anything;
    • policing developers and nagging them with your defects.
  • Testers do:
    • we ask questions – in order to investigate so we can make a decision;
    • we are information hunters;
    • “A tester is somebody who knows that things can be different” – Jerry Weinberg.
    • think in different perspectives:
      • user;
      • troublemaker;
      • stakeholder;
      • developer.
    • apply critical thinking (e.g. 6 thinking hats);
    • sometimes we use silliness;
    • we build relationships;
    • we use experience and irrationality;
    • we communicate, we tell stories and ask questions at the same time;
    • we coach, we mentor, we teach and we learn;
    • we make use of our senses (sad, angry, etc.);
    • we use our technical skills and our humour;
  • Testers are not:
    • a safety net (again!);
    • a gateway (again!);
    • a judge.

One thing to take into consideration is that even though testers are not judges, they could consider themselves as an expert witness at the trial. And we can’t judge quality because quality is relative from person to person.

The final slide from Tony’s presentation illustrates quite well what testers do, I recommend you give it a look!


“Two sides of the story” by David Evans and Brindusa Axon

David and Brindusa talked about user stories, the pros and cons and about using them and techniques for improving them.

  • The current path to stories is flawed, as we always end up biting off more than we can chew.
  • Good story characteristics:
    • INVEST
      • independent;
      • negotiable;
      • valuable;
      • “estimatable”;
      • small;
      • testable.
  • Typically however we lack the V for valuable.
  • The template for story stinks:
    • “Who?” there is almost never just one stakeholder;
    • “What?” what is the benefit of forcing an awkward style?;
    • “Why?” why does the last phrase always sound so weak?;
    • Use templates if it helps get the team going, but once you’re going going then you don’t need templates. Leave them behind.
  • Possible alternative for user story writing:
    • <verb> a <noun>
      in order to <improve quality>, we will do <some thing>
      in order to help <someone>
      achieve <some value>
      we will do <some thing>
    • plain language!
  • Liz Keogh’s blog post “They’re not user stories”
  • A story is a promise to hold a conversation – if you story card is too small to write everything, get yourself a small card. You don’t need all the details, have conversations instead.
  • Don’t let agile be an excuse to ignore documentation.
  • Don’t let documentation be an excuse to avoid a conversation.
  • Documentation: “like a lady’s skirt – long enough to cover the subjects, short enough to create interest”
  • Short lived (burn this):
    • story;
    • change;
    • acceptance criteria;
    • cards/tasks.
  • Long lived (keep this):
    • specification;
    • effect;
    • acceptance tests;
    • living documentation.
  • There’s no way you can say “I’ve written the perfect story”.
  • Signs of success with stories:
    • finding elegantly simple, not simplistic solutions;
    • Product Owner satisfied with doing less than intended;
    • transparent decisions on trade-offs;
    • fewer boomerangs and re-works;
    • highly collaborative creation process.
  • Collective product ownership idea in the team.


“BDD & The business analysts” with Kent McDonald, Jeffrey Davidson, Olaf Lewitz, Jake Calabrese, Leslie Morse and Christ Matts.

This talk was actually a conversation instead of your standard conference talk. Chris Matt was acting as the “host” and here’s a picture of what it looked like from the seats.


Here are some of the key points talked about:

  • Using BDD for scope definition (this was actually my first takeaway from BDD when I first used it, and I will at some point blog about it!).
  • BDD is a conversation and it should be easy – it’s how we interact with people!
  • There are 2 parts: work out what the questions are, and then find the answers.
  • You also need to structure it in a way so that the “conversation” doesn’t lose fidelity.
  • The “given when then” is necessary but not sufficient – conversations need to happen.
    • The necessity comes from offering consistency.
  • Real analysis still has to happen, just capturing stories isn’t enough as users may not be telling you what reality actually is.
  • If you’re only writing things down then you are not a business analyst, you’re a transcriber.
  • Mention of Liz Keogh again, and her blog post on “Estimating complexity”.
  • BDD focuses on behaviour and needs of stakeholders.
  • BDD is only of part of the “Analysis” toolkit, which is not a role.
  • We’re all doing BDD everyday (or should be doing), we just may not be writing it down and instead are hiding it from everyone.


“Successful collaboration and MOFF – Applied” by Chris Priest, Jenny Martin and Pete Buckney

They spoke about using the feedback onion (which apparently wants to become the feedback aubergine (less barriers between layers and won’t make you cry so much) and the MOFF collaboration framework.

MOFF – Maximise opportunities for feedback: opportunities, value and quality of interactions and using different techniques to achieve this, such as impact maps.

You can see the models here.