Why We Started This Experiment
We're convinced that in the coming years, prompt engineers (prompters) will partially — and eventually fully — replace many of the functions currently handled by traditional developers. As a tech company, we can’t afford to ignore this trend. On the contrary—we’re systematically integrating AI tool experiments into our workflows to stay ahead of the curve before it becomes “too late” for others.
What You'll Learn in This Digest
- How we approached selecting and testing AI tools for development
- What metrics we used to measure effectiveness
- What worked well—and what fell short
- Key takeaways and what’s next
Our Testing Approach
Six months ago, we launched an internal initiative to test AI tools in real-world projects. The goal wasn’t just to “play around,” but to understand where and how AI can already replace or enhance development today. Testing format:
- Each tool was given to 3–5 team members across various projects
- 75% of participants were mid-level developers
- 25% were senior developers
- Usage duration: 3 months
- All participants used paid versions of the tools
Evaluation Criteria:
- Task completion speed
- Reduction in bugs
- Decrease in escalations to senior developers
- Productivity trends over 3 months
Tools We Tested:
- ChatGPT (OpenAI, GPT-4)
- GitHub Copilot
- Codeium
- Claude AI (Anthropic)
- DeepSeek (AI model focused on programming)
Final Results (Post-Correction)
Tool | Task Speed | Bug Reduction | Fewer Senior Escalations | Productivity Increase (3 months) |
ChatGPT | +33.8% | -19% | -24% | +14% |
GitHub Copilot | +36.7% | -13% | -18% | +10% |
Codeium | +31% | -17% | -21% | +11% |
Claude AI | +31% | -22% | -29% | +17% |
DeepSeek | +31% | -14% | -18% | +8% |
Key Takeaways
Mid-level developers benefit the most
AI companions act as a “second brain” for mid-level developers — helping them understand unfamiliar code faster, handle routine tasks, and write tests more efficiently.
Senior developers use AI differently
Senior engineers don’t gain as much in speed but reduce cognitive load — spending less time on routine and more on high-level architecture and design.
Prompt engineering matters
Developers who completed even a basic workshop on working with AI performed 16–22% better than those who didn’t.
Claude AI is the best “AI reviewer”
It excels at explaining complex code, writing documentation, and “translating” legacy systems. While not always the fastest, it is often more accurate and clearer than others.
What’s Next?
- Expanding the use of Copilot and ChatGPT across core product teams
- Launching an internal course on AI-assisted development
- Integrating Claude AI into code review and documentation workflows
- Kicking off the next testing phase: AI in QA, marketing, analytics, and project management
A Note on Limitations
We fully recognize that our sample size was limited, and the projects varied in complexity. These numbers are not an absolute truth—they’re a working model for understanding change. But even with those limitations, the pattern is clear:
The AI revolution in software development is already happening.
What’s Coming Next
- How AI is transforming other departments inside Cody Solutions
- How team dynamics evolve under AI influence
- What new workflows emerge at the intersection of automation and development
- And how AI is progressing in code generation and code analysis
