In this article I would like to outline, at a high level, the overall thought process for “architecting” big data solutions. The process of developing an architecture is a very personal thing. I don’t propose to say that this is the only way – this is simply a framework from which you could launch your own effort.
The starting point for every architecture must be the business.
Some of the questions that should be answered are: “What business goal or goals am I trying to achieve with this initiative?” This question must give you the problem that you are trying to solve in the language of the business. Something like “What are people saying on the internet about product X in the last 30 days.” presents a clear gold that can be acted on.
“How will the solution data, process, and systems integrate into or change the business process? By defining these characteristics we can determine the effect the solution will have, any inconsistencies or conflicts with existing business processes. This activity will generate additional clarity for all involved.
“Have I confirmed that the proposed change is required, beneficial, and consumable by the business?” One of the questions that we don’t ask enough is if the change is going to benefit the company or is it just change for the sake of change. Many times we can develop a solution that an excellent technical solution but does not fit with the organization. Benefit generation and Organizational fit provide solutions that cause less resistance to change.