How effective are ChatGPT or other generative AI tools in performing architecture analysis to help better understand industry trends? I decided to try and find out by using ChatGPT’s new IOS app )launched in recent weeks) to learn about various technology offerings and industry trends. During my research, I asked the chatbot several questions and participated in various discussions to see what kind of feedback ChatGPT would give me. In the process, I uncovered five things for architects planning to use these tools for industry analysis to watch out for.
Overall, ChatGPT is a fantastic tool, offering me many insights into architecture industry tools and trends. However, there were some limitations, which are essential for architects to understand if they plan to use ChatGPT for external analysis.
- Different GPT models deliver different results. During my research, I tried GPT 3.5 and GPT 4.0 and obtained very different results from each. This indicates that these models are optimized differently and/or pulling from different underlying data sources. So I encourage architects to try various GPT models or even completely different AI bots. In my research, I asked questions in Bing, as well. When I asked the same questions, I also received different results. It appears the bots use various research sources. Regardless of the model or bot, none of the AI tools I experimented with did a great job aggregating all the insights. So be aware and use multiple AI sources to get your information.
- The data can be stale. Much of what I gleaned came with a warning from ChatGPT that the information was based on data as of September 2021. Using data at least 2 1/2 years old can be problematic in a rapidly evolving industry such as enterprise architecture. In my research, I discovered current products and trends from the last year weren’t present in the results. Using AI may not get you the latest data; what you get may be old and incomplete. So be aware of stale data.
- Results may not be specific to your organization or industry. These AI capabilities are excellent at addressing macro trends, and you can use prompts to filter it down or narrow information down by geography, region, or industry. However, the ability to effectively filter is limited. Architects should keep in mind that the responses they get may not apply to their industry. They may need further filtering or refinement to get answers specific to their organizations.
- Your questions are collected for training the next-gen AI models. Every question you ask is being used to train and improve future AI models. While contributing to AI is a good thing, you should also keep in mind with this comes the possibility of future bias. Be aware that these models are built upon collective queries, responses, (and shenanigans). Not everyone using ChatGPT or similar tools is asking questions for the same purpose, so replies are often skewed to meet the needs of the general public.
- Be careful about sharing your company’s sensitive information in your questions. Remember, with every question you ask or discussion you participate in, your information is collected by the AI tool. ChatGPT can write code, analyze databases, and do data analysis for you but with this power comes IP risk. If you’re asking specific questions relating to company-sensitive data or scenarios that could relate to strategic differentiation in your market, be aware that that information becomes public knowledge when you ask it. ChatGPT and other AI are public forums; you should treat them accordingly.
ChatGPT and other generative AI tools can be a great asset to help you understand external macro trends, but be sure to pay attention to the caveats and limitations of the data you’re receiving and the context of the responses. Use your own experience and other data sources to validate the answers you’re getting and the findings to ensure they are appropriate for your organization. And be sure as an architect to add your own perspective. As good as automated bots are, they can’t replace your experience, contextual awareness of your organization, and what problem you are trying to solve with the information.
