Skip to main content
Unlocking Property Management Insights: Extracting and Analyzing Yardi Data

Unlocking Property Management Insights: Extracting and Analyzing Yardi Data

Written by Cynthia Hoza on . Posted in Beyond Data Podcast, Data Analytics, Data Insights, Data Strategy, Data Visualization, Glossary, Resource Library. No Comments on Unlocking Property Management Insights: Extracting and Analyzing Yardi Data

 

Join Nick Mishko, Senior Data Analytics Team Lead at Calligo, as he delves into the world of property management analytics and Yardi data.

Discover how Calligo’s data analytics practice transforms Yardi data into powerful tools, enhancing operational efficiency for property management firms globally. From data extraction challenges to creating dynamic dashboards, explore the strategies and solutions that propel businesses forward.

If you’re navigating Yardi complexities or seeking to leverage analytics for your property management endeavours, this insightful discussion is a must-watch. Stay tuned for more insights from Calligo Shorts!

<< PREVIOUS EPISODE
NEXT EPISODE >>

Year in Review Video Title Slide Linkedin - 2MB Resize 02

Data Transformation Predictions for 2024 – Calligo Data Leaders Roundtable

Written by Cynthia Hoza on . Posted in Beyond Data Podcast, Cloud Migration, Cloud Services Insights, Cloud Strategy, Cyber Security, Data Analytics, Data Engineering, Data Governance, Data Insights, Data Privacy, Data Protection, Data Strategy, Data Visualization, Data Warehousing, Glossary, Machine Learning, Managed Cloud, Resource Library. No Comments on Data Transformation Predictions for 2024 – Calligo Data Leaders Roundtable

 

In this lively debate you will hear from Calligo’s Practice Leads as they discuss their key takeaways from 2023 and their data predictions for 2024 and beyond.

Topics discussed include:

Regulation of AI including the EU AI act

AI hallucinations & AI bias

Data governance and data fines

Dashboard fatigue

Data ROI

<< PREVIOUS EPISODE
NEXT EPISODE >>

The Data Management Conundrum: Data Lake vs. Data Warehouse with Calligo’s Warehouse as a Service

Written by Cynthia Hoza on . Posted in Data Engineering, Data Strategy, Data Warehouse, Data Warehousing, Glossary.

In the age of information, businesses are confronted with an unprecedented influx of data, making effective data management critical for success. Two prominent solutions have emerged to address this challenge: data lakes and data warehouses. Each offers distinct advantages and use cases, catering to the diverse needs of modern enterprises. In this comprehensive exploration, we’ll dive into the fundamental differences between data lakes and data warehouses, and then we’ll shine a spotlight on Calligo’s Warehouse as a Service (WaaS) solution as a forward-thinking approach to data warehousing.

Data Lake vs. Data Warehouse: Navigating the Terrain
Data Lake: The Uncharted Waters

A data lake is a vast repository that can store structured, semi-structured, and unstructured data in its raw form. This makes it an ideal solution for organizations dealing with diverse data types and sources. Technologies like Apache Hadoop and Apache Spark are commonly associated with data lake implementations. Key strengths of data lakes include:

Flexibility: Data lakes accommodate raw and unstructured data, allowing organizations to ingest information without the need for predefined schemas.
Scalability: Built to handle massive data volumes, data lakes scale horizontally, making them well-suited for big data analytics.
Cost-Effective Storage: Storing raw data in a data lake is often more cost-effective compared to the structured storage in a data warehouse.
Data Warehouse: The Organized Harbor

In contrast, a data warehouse is a structured repository optimized for efficient querying and analysis. It stores data from various sources in a predefined, tabular format, enabling quick access for reporting and business intelligence activities. SQL databases are commonly used in data warehouse implementations. Key strengths of data warehouses include:

Structured Querying: Data warehouses excel in structured data querying, providing rapid access to organized information.
Performance: Aggregated and pre-processed data in a data warehouse enhances query performance, making it ideal for complex reporting and analytics.
Data Quality: Data warehouses enforce governance and quality standards, ensuring reliable and consistent data.

Calligo’s Warehouse as a Service (WaaS) Solution: Navigating Both Worlds
Amidst the dichotomy of data lakes and data warehouses, Calligo’s Warehouse as a Service (WaaS) solution emerges as a beacon of innovation, seamlessly integrating the strengths of both paradigms. This holistic approach empowers organizations to leverage the benefits of both data lakes and data warehouses within a unified platform. Let’s delve into the key features that make Calligo’s WaaS a game-changer:

  1. Unified Platform:
    Calligo’s WaaS bridges the gap between data lakes and data warehouses, providing a unified platform for holistic data management. It allows organizations to store raw data in a flexible and cost-effective data lake while maintaining a structured and optimized subset in the data warehouse for analytical purposes. This integration enhances agility and ensures that the right data is available for the right purpose.
  2. Optimized Storage:
    One of the distinctive features of Calligo’s WaaS is its intelligent storage management. Raw data can be stored in its native format within the data lake, minimizing costs associated with storage. Simultaneously, a curated and optimized subset of the data is stored in the data warehouse, ensuring high-performance analytics without compromising on the advantages of a data lake.
  3. Advanced Analytics:
    Calligo’s WaaS is equipped with powerful analytics capabilities, enabling organizations to derive actionable insights from their data. The platform supports complex reporting, data visualization, and business intelligence, providing decision-makers with the tools they need to make informed choices.
  4. Data Governance:
    Recognizing the paramount importance of data governance, Calligo’s WaaS prioritizes compliance with regulatory standards and maintains data quality across the entire data lifecycle. This ensures that organizations can trust the integrity and reliability of their data, fostering a culture of responsible data management.

Conclusion: Navigating the Data Landscape with Calligo’s WaaS
In the evolving realm of data management, the choice between a data lake and a data warehouse is often a complex decision based on specific organizational needs. Calligo’s Warehouse as a Service solution transcends this binary, offering a unified platform that integrates the best of both worlds. By seamlessly combining the flexibility of a data lake with the structured efficiency of a data warehouse, Calligo’s WaaS emerges as a pioneering solution for businesses seeking to navigate the complexities of modern data management. As organizations strive for data-driven excellence, the synergy of data lakes, data warehouses, and innovative solutions like Calligo’s WaaS can pave the way for a more efficient and insightful future.


For more comprehensive insights into data warehouse strategy, visit https://cal.essence-design.co.uk

esg

Powering up ESG through digital transformation

Written by Cynthia Hoza on . Posted in Data Ethics, Data Insights, Data Strategy, Glossary. No Comments on Powering up ESG through digital transformation

The term ‘ESG’ (Environmental, Social and Governance) is everywhere. In its own right, the potential impact is important enough, but it can so often be viewed as a standalone initiative. At its worst it becomes a tick box exercise, when in fact its real benefit is in informing and driving fundamental changes in your organization’s wider actions and endeavors.

ESG – good for the planet, good for business

In January 2023, the EU’s Corporate Sustainability Reporting Directive came into effect. Under its terms, all large companies and all listed companies (except micro-enterprises) must disclose information on the risks and opportunities arising from social and environmental issues, and their impact on people and the environment.

Set against this we have an AI revolution taking place – witness the activity on LinkedIn, with almost every other post lauding the benefits of some ChatGPT derivative or similar, leading to something of an AI feeding frenzy.

Looking through an ESG lens, the environmental impact of AI is huge. According to calculations by the specialist in sustainable data science, Kasper Groes Albin Ludvigsen, published in Medium at the end of 2022, ChatGPT could have consumed as much electricity as 175,000 people in the month of January 2023 alone. Equally, there are numerous articles that reference AI’s huge water impact.

One thing is clear. Whilst there can be many positive outcomes and by products from AI on ESG, the true end-to-end cost of this next wave of Digital Transformation is not yet well understood.

Given we are still trying to get to grips with the effects of the Industrial Revolution from an environmental perspective, how good is humankind’s track record of not repeating the mistakes of the past? How can we exploit opportunity without understanding the true cost and impact?

Wider business benefits of ESG

Developing an ESG strategy that is in harmony with your Digital Transformation yields multiple advantages. And whilst ESG reporting is now mandatory for corporations in the EU, doing so helps quantify the benefits that exist for every party:

  • Investors. Many investors place great importance on ESG reporting and an overall strategy
  • Customers. Consumers are increasingly concerned about the companies they place business with, and ESG is becoming far more important in their decision making
  • Suppliers / Supply Chain. Companies are receiving more requests for information on their ESG credentials, capabilities and response. They must be able to demonstrate their end-to-end position when reporting, driving positive change throughout the supply chain
  • Employees. Recruiting and retaining talent can be difficult, expensive and disruptive when there are issues with ESG policies. Research indicates that as many as 47% of employees would look for new roles if their organization is not proactive here
  • Market reputation. Creating a strong reputation and a positive view of a company takes time and effort. Negative disclosures around ESG will quickly damage reputations, whereas positive ones will confer competitive advantage

Balancing potential conflicts between digital transformation and ESG

Detractors of ESG will point to the irony that a robust ESG process itself has an environmental impact: data centers in the EU consume more than 2.7% of the bloc’s electricity. And the Ukraine war has highlighted that the geopolitics of power supply will increasingly affect decisions on data processes and sovereignty – when Cloud storage and transference requires so many terawatts of electricity, securing a good price must be balanced against political and geographic risk.

Digital transformation is, by its very definition, a process of huge change. Done right it unlocks competitive advantage, delivers cost savings, drives productivity, opens up new opportunities and delivers compliance with ESG obligations. But done half-heartedly or implemented sporadically it will almost certainly be a huge waste of time, effort and resources.

Deloitte calculates that digital transformation could unlock as much as US$1.25 trillion in additional market capitalization across all Fortune 500 companies. However, done incorrectly, market value could actually be eroded, putting more than US$1.5 trillion at risk.

Prior preparation prevents poor performance

When it comes down to it, successful digital transformation requires only three things:

  • An agreed plan
  • The right tech platforms
  • A joined-up approach

And whilst that sounds simple, it involves significant planning and project management resources. It’s not possible to retro-fix a digital solution onto your existing processes – a successful digital transformation requires a center-out approach, incorporating data privacy and protection and considering ESG objectives at the very heart of policy and technology.

When digital transformation is done correctly, “it’s like a caterpillar turning into a butterfly,” but when done wrong, “all you have is a really fast caterpillar.”

MIT Sloan Professor George Westerman

ESG at the heart of the digital transformation process

The comprehensive and insightful data analysis and management required to power your digital transformation needs a huge team of business experts, platform designers and technology specialists, all following a clear process:

  • Develop an agreed, business-wide strategy
  • Create and share a roadmap
  • Define the metrics of success, and measure them
  • Build user-friendly dashboards and data analytics
  • Use optimal data platforms and cloud services
  • Ensure data privacy and protection
  • Set and track ESG targets. Not only does ESG need to be considered, it needs to sit right at the heart of digital transformation, informing and guiding the entire organization


Simply ‘ESG washing’ operations with fancy reports is both ineffective and expensive. That’s why Calligo ensures that every digital transformation we drive is engineered with careful attention to its environmental impact. Future-proofing your data use in a way that protects everyone’s future.

To help you navigate the expansive topic of digital transformation, we’ve put together a comprehensive eBook, outlining all the key considerations for your organization. And if all this sounds daunting, don’t worry –  we’ve seen plenty of similar challenges. Data privacy, for example. Once seen as a vague afterthought or something for someone else, today it takes center stage – the concept of Privacy by Design even has its own ISO standard (31700). Understanding the end-to-end ESG impact of Digital Transformation is heading the same way.

If you want to learn some more, or if you want specific advice, consultancy support or technical implementation, why not talk to our experts, who can get your digital transformation journey underway?

complex data

Making complex data available for the benefit of society

Written by Cynthia Hoza on . Posted in Beyond Data Podcast, Data Governance, Data Insights, Data Privacy, Data Privacy, Data Strategy, Data Visualization, Glossary, Machine Learning. No Comments on Making complex data available for the benefit of society

Listen on Spotify

In Calligo’s latest Beyond Data podcast, Tessa Jones (Chief Data Scientist) is joined by Dr Ellie Graeden, Research Professor (Center for Global Health Science and Security) at Georgetown University. Here we explore some of the episode’s highlights:

  • The inherent conflict of private data and the public good
  • Protecting individual rights within federated learning
  • The importance of effective communication and a common language
  • Designing systems and policies that work together
  • Focusing regulation on outcomes, not creating data siloes

At societal level, poor communication costs lives

Transitioning data across and between departments and data systems has historically been fraught with problems – who owns it? Who pays for it? Is it understandable and translatable into meaningful and actionable insights for the end user? 

Having worked extensively in disaster response, Dr Graeden has seen first-hand the potentially life-threatening issues that can arise when government departments’ data platforms produce incompatible outputs:

  • If 20,000 people need water, how many pallets need to be shipped?
  • If 10,000 electricity meters have been knocked out by a hurricane, how many people need feeding?

In such scenarios, identifying individuals amongst population-level data is crucial if the help provided is to be sufficient.

“We have to be able to really effectively move and communicate and share data that are relevant, in ways that they can get used by people all across the system”

Of course, any data system design should ensure privacy and protection for personal data. ‘Big data’ is still relatively new, and as such more powerful and widespread regulatory controls are now being introduced, although the US still does not have consistent requirements for how data should be handled. Fundamentally, meeting a population’s needs today, and planning for them tomorrow, requires the data of individual people to be analysed. Personal data must be shared quickly, effectively and all the while protecting individual rights. Data system design must therefore:

  • Include all players
  • Consider cultural constraints
  • Keep out bias
  • Ensure the right words and phrases are used
  • Focus on the ‘so what’, why does it matter?

“Every single thing we experience can be captured as data”

Even the most mundane moments in our daily lives leave a digital footprint, we shed data everywhere. But when does ‘my’ data become public, or the property of the software developer or the service provider? VR headsets collect ephemeral data that is analysed and applied for that one end user, but if that data is assumed to fall under GDPR the potential to use it for positive outcomes is severely limited. For example, should authorities be notified if content viewed and generated is illegal or harmful? And what if that chip can detect if the user is having a stroke, is that data classified as ‘health’ data? Can it be used to alert the individual to their medical emergency without contravening legislation? What if your mouse clicks can detect the early stages of Parkinson’s? Should you, could you, be told?

“If you’re treating this data as health data, then they have a very different set of regulatory constraints. HIPAA isn’t going to regulate those because it’s not a health care provider or a health insurer”

Piercing the veil

The conflict between personal protection and public good is everywhere, and Dr Graeden believes that some new data laws will create problems for federated learning. Legislation has clear boundaries (speed limits, blood alcohol levels) whereas science deals in spectrums, probabilities and unknowns.

Deleting an individual’s personal data from the model breaks the system, contradicting what regulators are trying to achieve. The solution is to prioritize outcomes, not processes – it doesn’t matter whether you write the rules with a pen and paper, or with AI, as long as you write the rules. Expanding the framework by setting gradients of data availability affords protection for individuals, whilst making data available that informs better decision making for public bodies.

“Data is nothing more, nothing less, than an abstract description of our world. A useful and powerful language that can tell us things that other languages don’t”

Data can no longer exist in siloes if it’s to be useful to society

There is now a healthy global appetite for the discussion around data, thanks in the main to two recent developments:

  • Covid gave us huge amounts of data about mortality levels, vaccination rates, hospitalisation trends – all of which were in the public consciousness every day
  • AI and ChatGPT – articles and debates about the pros and cons are everywhere, discussion is not just in the scientific community

The key challenges now for data scientists are expectation management and communication – we need to be clear about aims and specific about context, as well as knowing what to leave out to avoid overwhelm and misunderstanding. Unfortunately, scientists are not always great communicators (using complex terminology and detail, rather than common parlance and generalization) as Covid demonstrated:

  • Did having a vaccine mean you wouldn’t get sick? Or just less sick?
  • ‘Everyone should wear a mask’ became ‘wear a mask if you can’. This was due to limited supply, but it appeared that the science was not clear

“The scientific approach means you never have an answer… we are trained as scientists to focus on the fact that we don’t know”

In fact, the only answer is that the right data, used consistently and communicated clearly, will always allow us to be prepared, not reactive. To make decisions for the public good that protect every individual.

You can find out more about the common language of privacy in our Rosetta Stone eBook.

You can also watch Tessa’s fascinating podcast with Dr Graeden below.

<< PREVIOUS EPISODE
NEXT EPISODE >>

The dark side of AI energy consumption – and what to do about it

Written by Brendan Walsh on . Posted in Beyond Data Podcast, Data Ethics, Data Privacy, Data Strategy, Glossary, Machine Learning, Machine Learning. No Comments on The dark side of AI energy consumption – and what to do about it

Listen on Spotify

Artificial Intelligence’s ability to augment and support progress and development over the past few decades is inarguable. However, when does it become damaging, contradictory even? In our latest Beyond Data podcast AI’s Climate Jekyll & Hyde – friend and foe, Tessa Jones (our VP of Data Science, Research & Development) and Sophie Chase-Borthwick (our Data Ethics & Governance Lead) discuss exactly this with Joe Baguley, Vice President and Chief Technology Officer, EMEA, VMware.

Our speakers explore the multifaceted topic of energy consumption and AI – from whether all applications are equal for energy consumption (or reflecting if there are some ‘better’ than others), to creating visibility and responsibility of energy consumption for all stakeholders. Here we try to give clarity to some of the grey areas that were discussed.

Should we consider all applications equal?

“AI and machine learning are about huge things, huge data sets, huge computation actions … all of those have huge implications in terms of energy,” Joe observes, before dropping in hugely sobering stats such as the total annual energy consumption of bitcoin being the same as Norway. Even when considering the often-touted argument of 57% of the energy for bitcoin mining using renewables, Joe counters: “But those renewables could have been used for something else, right? Those solar panels… and those hydropower stations and those wind turbines, we could be using them for something else.”

This raises the ethical question of whether there should be stricter governance, standards, and precedent set on more ‘moral’ applications for their energy consumption. Should we be more closely considering the difference in energy consumption between server farms that support minimizing food waste versus those that are focused on mining digital currency, for example?

“Is there an opportunity for [greater] regulation?” Tessa ponders. Would this regulation help challenge the current status quo for all applications’ energy consumption being considered equal? While Sophie observes: “We’ve had certain European nations start to put rules around data center expansion, where you’re allowed and not allowed to build because of the capacity there, which isn’t regulating the use of it. But it does have that knock-on effect that if you literally can’t build the data center support, you have to start thinking about other ways to build [models].”

When considering Sophie’s point on alternative ways to build models, Joe notes: “We’re using AI to deal with the symptoms, but maybe there’s some better ways we could be using AI to deal with the cause as well”.

And this all raises the next question – who should ultimately be making these ongoing moral calls for the environment and energy usage?

Embedding Environmental, Social, and Governance (ESG) by design

Environmental, Social, and Governance (ESG) is shorthand for a framework that helps stakeholders understand how an organization is managing risks and opportunities related to environmental, social, and governance criteria. Our speakers untangle the idea of ESG and how companies could use it to help triage the different applications they use.

Joe asks: “Is there an ESG-led marketing opportunity here? Your AI might be the same as my AI, but my AI is better from an ESG perspective. They both get the same results at the same time for the same cost, but this one’s better from an ESG perspective, in terms of sustainability, in terms of social good, in terms of environmental.”

By placing more emphasis on ESG as the criterion for measuring impact and success, it could help with embedding sustainability in the heart of the application’s deployment, rather than a siloed approach. Sophie agrees: “We have privacy by design, we have security by design. Why not have ESG by design?”

Following on from this thought, our speakers consider the cost implications of AI and ESG with Joe observing, “There’s a lot of businesses right now that can’t afford AI because it’s expensive…but I believe they will come to a tipping point where they can’t afford not to”.

Are we over-prioritizing accuracy?

“There’s a hyper-focus on the accuracy,” according to Tessa. “It ends up not even being about the motivation for green, it’s a motivation for fast training, fast tuning. Unfortunately, it’s how most data scientists are motivated; be faster without having to compromise their accuracy.”

Often, the increase in accuracy can be mapped on a logarithmic graph. Good gains at first, but quickly tapering off to minimal increase. Is it useful to be that much more accurate, often by points of a decimal? “Some are good, more must be better … people just keep going, as opposed to saying actually good enough is good enough,” Joe summarizes.

Instead of chasing marginally better accuracy each time, we should be considering the application in a holistic view. The increase in accuracy might be 0.01%, but would cost heavily for energy consumption – is it worth it? Should we be better at exposing these costs more vigorously throughout a team so everyone can feel more empowered and have the visibility to interrogate more closely?

To hear about how our speakers untangle these controversial questions and more, tune in now to Beyond Data podcast episode 3: AI’s Climate Jekyll & Hyde – friend and foe.

<< PREVIOUS EPISODE
NEXT EPISODE >>


Raconteur Special Report – Digital Transformation

Written by Brendan Walsh on . Posted in Data Strategy, Glossary. No Comments on Raconteur Special Report – Digital Transformation

Calligo features in Raconteur’s latest special report on Digital Transformation, which was published in The Times.

The 16-page report contains insights from recognised experts and thought leaders on Digital Transformation strategy and execution and looks at how coronavirus has accelerated digital transformation in businesses and education. The report also explores the future of work and innovation, and why data-driven approaches are key to successful digital transformation projects.

“97 per cent of business decision-makers say that COVID-19 pandemic has sped up digital transformation at their company”

Raconteur

Calligo was asked to contribute on how best to design a Digital Transformation project, and how to ensure its immediate and ongoing success.

Our article, “Five steps to successful digital transformation”, featured on page 5 of the Raconteur report, provides a how-to guide to building a modern, data-first Digital transformation strategy that delivers.

The steps include:

Why you should not start with your business needs
The benefits of adopting a privacy-first mindset
How to build a strategy that earns your customers’ trust and makes them enjoy working with you
When and how to deploy technology
Why your Digital Transformation strategy has to keep evolving, and how to maintain it

“If you take a technology-first approach to Digital Transformation, it relies on identifying business problems and deploying the most suitable technology to fix them.

“In contrast, a data strategy starts by examining how data moves through the business, identifying areas of inefficiency, data governance weaknesses, overspend, security gaps and so on.

“It is a far more fundamental approach to Digital Transformation, improving businesses from their very foundations and bringing more value as a result.”

Adam Ryan, Chief Services Officer

Data Privacy and Data Security Recommendations for COVID-19

Written by Brendan Walsh on . Posted in Data Privacy, Data Strategy, Glossary, Managed Cloud. No Comments on Data Privacy and Data Security Recommendations for COVID-19

The speed that COVID-19 spread around the globe and the lockdown that followed has caught many companies off guard, and there’s a good chance that you may even be reading this in a hastily-assembled home office, in your kitchen or a spare bedroom.


For some, the ability to keep data secure has been torpedoed by unexpected, sudden volumes of employees working from home, relying on domestic networks and personal devices. Similarly, there has been widespread confusion over how to balance employees’ privacy and confidentiality with the broader obligation of staff protection and even civil responsibility.


Navigating these times is difficult but there is some comfort in knowing that even during this emergency situation, the normal rules still apply.


Our Data Privacy team has released new guidance on the Data Security and Data Privacy concerns of the ‘new normal’, in order to help businesses follow data privacy rules and security best practice, while protecting the health and preserving the productivity of their staff.