임베디드 소프트웨어의 Top 5 문제
대체로 application영역이라기 보다는 embedded system영역이라 general software만 해왔던 사람들이 다루기 힘든 영역이 아닐까 생각한다.
- 코드 복잡도는 ?
- 실시간성 만족여부
- 소프트웨어 품질의 충분성 여부
- 프로세스 방법론이 충분히 엄격한가?
- dependability 관점 ?
1) 코드 복잡도에 대해
여러가지 지표가 있고 분명 유용하다고 판단되는 지표 및 기준이 존재한다. 문제는 그것을 지키려고 하지 않는 소프트웨어 개발자들과, 품질관점의 복잡도 제어에 관심이 없는 최종 수요자이다.
분명 software metrics는 직접적으로 제품의 오작동에 영향을 끼친다는 것을 입증할 수 있는 증거가 있는 것은 아니다. 그러므로 필수적으로 해야 할 필요가 있나? 란 의문이 생긴다. 게다가 이런 부류들은 대체로 정량적으로 평가할 수 없는 것들이라서 그 수준에 대한 validity를 보이기 쉽지 않다는 것이 문제이다.
2) 실시간성 만족여부
임베디드 시스템이 고성능을 위해 설계된 구조로 인해 시스템의 실시간성에 대한 부합여부를 판단하기 위해 어려운 점들이 있다. (Cache, page 등)
non determinism적 요소들은 시험으로는 요구사항을 입증할 수 없으므로 Verification이 꽤 까다롭다.
그렇다고 쥐잡는데 소잡는칼 쓸수도 없고(과대 비용)…
가끔 이유없이 문제생기는 것들이 이런 원인이 될 수 있다.
3) 소프트웨어 품질의 충분성 여부
Most severe하게 시스템을 개발하는 프로세스에 대해서는 뭐~ 알겠다.
근데, 어중간한 수준에서 시스템을 개발하려고 하면 어느 수준까지 개발해야 할까?
표준에서도 나와있는 내용이긴 하지만, 칼로 무자르듯이 명쾌하게 정의되어 있지는 않다.
Reference: Top 5 Embedded Software Problem Areas
Several times a year I fly or drive (or webex) to visit an embedded system design team and give them some feedback on their embedded software. I’ve done this perhaps 175 times so far (and counting). Every project is different and I ask different questions every time. But the following are the top five areas I’ve found that need attention in the past few years. “(Blog)” pointers will send you to my previous blog postings on these topics.
How does your project look through the lens of these questions?
(1) How is your code complexity?
- Is all your code in a single .c file (single huge main.c)?
- If so, you should break it up into more bite-sized .c and .h files (Blog)
- Do you have subroutines more than a printed page or so long (more than 50-100 lines of code)?
- If so, you should improve modularity. (Blog)
- Do you have “if” statements nested more than 2 or 3 deep?
- If so, in embedded systems much of the time you should be using a state machine design approach instead of a flow chart (or no chart) design approach. (Book Chapter 13). Or perhaps you need to untangle your exception handling. (Blog)
- If you have very high cyclomatic complexity you’re pretty much guaranteed to have bugs that you won’t find in unit test nor peer review. (Blog)
- Did you follow an appropriate style guideline and use a static analysis tool?
- Do you limit variable scope aggressively, or is your code full of global variables?
(2) How do you know your real time code will meet its deadlines?
- Did you set up your watchdog timer properly to detect timing problems and task death?
- Do you know the worst case execution time and deadline for all your time-sensitive tasks?
- Just because the system works sometimes in testing doesn’t mean it will work all the time in the field, whether the failure is due to a timing fault or other problems. (Blog)
- Did you do scheduling math for your system, such as main loop scheduling?
- Did you consider worst case blocking time (interrupts disabled and/or longest non-context-switched task situation)?
- If you have one long-running task that ties up the CPU only once per day, then you’ll miss deadlines when it runs once per day. But perhaps you get lucky on timing most days and don’t notice this in testing. (Blog)
- Did you follow good practices for interrupts?
(3) How do you know your software quality is good enough?
- What’s your unit test coverage?
- If you haven’t exercised, say, 95% of your code in unit test, you’re waiting to find those bugs until later, when it’s more expensive to find them. (Blog) (There is an assumption that the remaining 5% are exception cases that should “never” happen, but it’s even better to exercise them too.)
- In general, you should have coverage metrics and traceability for all your testing to make sure you are actually getting what you want out of testing. (Blog)
- What’s your peer review coverage?
- Are your peer reviews finding at least 50% of your defects?
- Does your testing include software specific aspects?
- A product-level test plan is pretty much sure to miss some potentially big software bugs that will come back to bite you in the field. You need a software-specific test plan as well. (Blog)
- How do you know you are actually following essential design practices, such as protecting shared variables to avoid concurrency bugs?
- Are you actually checking your code against style guidelines using peer review and static analysis tools? (Blog)
- Do your style guidelines include not just cosmetics, but also technical practices such as disabling task switching or using a mutex when accessing a shared variable? (Blog) Or avoiding stack overflow? (Blog)
(4) Is your software process methodical and rigorous enough?
- Do you have a picture showing the steps in your software and problem fix process?
- If it’s just in your head then probably every developer has a different mental picture and you’re not all following the same process. (Book Chapter 2)
- Are there gaps in the process that are causing you pain or leading to problems?
- Very often technical defects trace back to cutting corners in the development process or skipping review/test steps.
- Skipping peer reviews and unit test in the hopes that product testing catches all the problems is a dangerous game. In the end cutting corners on development process takes at least as long and tends to ship with higher defect rates.
- Are you doing a real usability analysis instead of just having your engineers wing it?
- Do you have configuration management, version control, bug tracking, and other basic software development practices in place?
- You’d think we would not have to ask. But we find that we do.
- Do you prioritize bugs based on value to project rather than severity of symptoms? (Blog)
- Is your test to development effort ratio appropriate? Usually you should have twice as many hours on test+reviews than creating the design and implementation
- Time and again when we poll companies doing a reasonable job on embedded software of decent quality we find the following ratios. One tester for every developer (1:1 head count ratio). Two test/review hours (including unit test and peer review) for every development hour (2:1 effort ratio). The companies that go light on test/review usually pay for it with poor code quality. (Blog)
- Do you have the right amount of paperwork (neither too heavy nor too light)
- Yes, you need to have some paper even if you’re doing Agile. (Blog) It’s like having ballast in a ship. Too little and you capsize. Too much and you sink. (Probably you have too little unless you work on military/aerospace projects.) And you need the right paper, not just paper for paper’s sake. (Book Chapters 3-4)
(5) What about dependability aspects?
- Have you considered maintenance issues, such as patch deployment?
- If your product is not disposable, what happens when you need to update the firmware?
- Have you done stress testing and other evaluation of robustness?
- If you sell a lot of units they will see things in the field you never imagined and will (you hope) run without rebooting for years in many cases. What’s your plan for testing that? (Blog)
- Don’t forget specialty issues such as EEPROM wearout (Blog), time/date management (Blog), and error detection code selection (Blog).
- Do your requirements and design address safety and security?
- Maybe safety (Blog) and security (Blog) don’t matter for you, but that’s increasingly unlikely. (Blog)
- Probably if this is the first time you’ve dealt with safety and security you should either consult an internal expert or get external help. Some critical aspects for safety and security take some experience to understand and get right, such as avoiding security pittfalls (Blog) and eliminating single points of failure. (Blog)
- And while we’re at it, you do have written, complete, and measurable requirements for everything, don’t you? (Book Chapters 5-9)
For another take on these types of issues, see my presentation on Top 43 embedded software risk areas (Blog). There is more to shipping a great embedded system than answering all the above questions. (Blog) And I’m sure everyone has their own list of things they like to look for that can be added. But, if you struggle with the above topics, then getting everything else right isn’t going to be enough.Finally, the point of my book is to explain how to detect and resolve the most common issues I see in design reviews. You can find more details in the book on most of these topics beyond the blog postings. But the above list and links to many of the blog postings I’ve made since releasing the book should get you started.