Two new rules flagged by the Australian Energy Market Commission overnight herald the start of fundamental changes to the way Australia’s energy markets operate, and a belated switch away from the centralised fossil-fuel network to a smarter, cleaner, cheaper and more reliable grid of the future.

In a sign that the penny may finally have dropped for the conservative energy market rule maker – under intense pressure from state energy ministers, consumers and analysts – the AEMC’s new rules address the two biggest cost centres for electricity consumers: networks and wholesale markets.

And they recognise the increasing role that new technologies such as battery storage, localised renewable energy and demand response will play in the market, and in the transition away from highly polluting, and increasingly expensive, fossil fuel generation.

With networks, which account for nearly half the cost of household bills, the AEMC has finally recognised that building more poles and wires should no longer be the default position of network companies, so it has moved to make it easier for them to consider distributed energy, which means localised renewables and storage, as a cheaper alternative.

“Developments in battery storage, demand response and other technologies may provide alternatives to ‘poles and wires’ when parts of the network need to be replaced,” it said in its draft ruling.

This will likely help pave the way for network owners such as Western Power to break down their “centralised” grid and break it into a modular network of micro grids, mostly renewables based and either stand alone or connected with a “thin wire” to the main network. Operators in Queensland, NSW, Victoria and South Australia are expected to follow suit.

Even so, there is some skepticism amongst energy analysts about whether these changes go far enough, and introduce enough competition to the market. They say their impact may demand on decisions by the Australian Energy Regulator. But it is considered to be a step in the right direction.

In wholesale markets, which account for around one third of the costs, the change is more profound – the adoption of a “5-minute” settlement period that will encourage faster, smarter and cleaner alternatives such as battery storage – paired with renewables – to compete with the relatively slow and clunky gas-fired generators in responding to changes in demand, supply and voltage.

The AEMC has been mulling this proposed rule change for nearly 18 months, and even though it couched the changes in terms of needing a rapid response to the “intermittency of wind and solar”, it appears it has finally accepted that the fossil fuel oligopoly has been rorting the wholesale markets, and pushing up costs to consumers through their bidding practices within the current 30 minute period.

The rule change had been proposed by Queensland zinc producer Sun Metals, which had become sick of the price gyrations (mostly up) caused by the bidding practices of the fossil fuel generators.

However, unlike the dramatic change to forex markets, where the Australian currency was floated overnight, the AEMC proposes in a “directions” paper that the 5-minute rule change be introduced over a three year period.

This is both a recognition by the AEMC of the potential complexities and costs of the changes to metering and IT systems, and possibly of the fossil fuel generators’ threat that the lights may go out if any change is too abrupt.

Such threats have been the not-always-unspoken default position of the fossil fuel generators on virtually every change in policy and rules – from the carbon price, through to the renewable energy targets, energy efficiency, demand response and emission limits.

In this case, the oligopoly suggested the costs would be so great that it would force generators out of the market. Such threats should normally be treated with absolute contempt, but as the South Australia government found out to its cost in February, some generators can simply not provide power, leading to blackouts.

In the case of the settlement period, despite huge objections from the main generating companies, the AEMC has decided that the benefits of the 5-minute rule change outweighs the costs.

A move to five minute settlement would align the physical electricity system – which matches demand and supply every five minutes – with the price signal provided by the spot market for that five minute period. The only reason it ever had a 30 minute settlement period was because of the limitations in metering and data processing in the 1990s.

But this 30-minute settlement period has opened itself up to rorting by the big generators, who have been shown to withdraw capacity and push up prices in one five minute bidding period, usually the first or second in a period, knowing such tactics would guarantee a high price for the entire 30 minute period.

Capacity then floods the market as generators “pile in” – as the AEMC described it – “to share the benefit of the high price event.”

It provided as an example this event just a few weeks ago, on March 21, in South Australia, where over five dispatch periods the price was inflated in the first bidding period (and one second bidding period) of each 30-minute interval.

“This bidding behaviour has the potential to significantly distort operational, usage and investment incentives, creating productive, allocative, and investment inefficiency,” the AEMC said. “It is difficult to reconcile a framework that contributes to such behaviour as being consistent with the NEO.”

(This refers to the National Electricity Objective which still, outrageously, does not include the environment)

Indeed, the costs of the bidding practice is illustrated in this graph below. In the first interval, the average cost of generation over those five periods is nearly $10,000/MWh, while in the second period is is $1,538/MWh. As the slow gas generators finally get their machines into gear to get a “slice of the action”, the prices slide into negative territory.

In short, it recognises that most of the price spikes witnessed in the electricity market in recent years are not the cause of a supply shortfall, or of renewable energy impacts as many would have you believe – but of deliberate acts by generators to artificially inflate prices. Consumers are now paying the price of this market sabotage.

It is difficult to imagine a more blatant sort. But there have been plenty of them.

AGL even admitted to the ability of generators to force “random” price spikes, while arguing that many peaking gas plants would be too slow to take advantage in a 5-minute settlement. “If such spikes were sufficiently random, this may force fast start generators to exit the market – as they may no longer be capable of generating a sufficient revenue to remain viable.”

The AEMC acknowledges that a 5-minute settlement would provide a more accurate signal that would lower prices and make the market more efficient, and encourage new technologies such as battery storage and demand response.

“(The 30-minute settlement) can also lead to bidding behaviour and operational decisions that result in responses occurring up to 25 minutes after they are needed by the power system,” the AEMC wrote.

“This could increase the cost of supplying electricity in the short and long term. There is some evidence of this occurring in the market today …. a more efficiently functioning wholesale market will in turn provides the benefits of lower supply costs and lower retail prices for consumers.”

Analysts and promoters of new technologies expressed their support for the new rules, albeit with caveats. With the 5-minute rule, they wondered how the AEMC arrived at a 3-year transition period, although they recognised the costs involved. It is understood some of the major fossil fuel generators argued for a 5 or 10 year transition.

Ross Garnaut, chairman of Zen Energy, a battery storage provider that has pushed for these changes, said there is no need for such a long transition.

“Markets adjust quickly to changes of greater dimension than would be required by the introduction of 5 minute settlement,” he said in an emailed statement to RenewEconomy.

“The proposed long transition would slow the introduction of new technologies that are already making large contributions to grid stability in other developed countries. To allow three years to pass before introducing 5 minute settlement, would unnecessarily expose Australians to insecurity in power supply over a challenging period. ”

The Greens also argued that the change could happen immediately. “This change is long overdue. The process has been glacially slow and it has been holding back more storage coming online. Another 3 year wait is not acceptable,” energy spokesman Adam Bandt said.

“We must guard against complacency. The incumbent fossil fuel generators will fight this rule because they want to keeping gaming a system that is set up to allow them to profit at the public’s expense. They will want to stretch a transition for as long as possible.”

Reaction to the network ruling was more varied, with some accepting it as welcome move forward, though dependent on other initiatives, and others saying it did little or nothing to address the basic regulatory issues around network spending.

Hugh Grant, the executive director of ResponseAbility, and a long-time critic of excessive network spending, said he welcomed the 5-minute rule, but described the network adjudication from the AEMC as “far too little too late”.

“It is highly unlikely that any real positive outcomes will arise from such incremental changes to the current regulatory arrangements,” he said in emailed comments.

Grant said the governance arrangements of the regulatory test (known as RIT-D) and the AER’s anticipated demand management framework are “fundamentally flawed” as they continue to provide the “networks with the power to decide whether non network solutions are deployed, and will continue to provide the networks with windfall profits for ‘business as usual’ activities.”

“In the absence of meaningful reform, Australia’s energy consumers should place more faith in technology and market forces, rather than regulation, to address the systemic bias in the current regulatory framework.”

The AEMC is inviting submissions to its directions paper on the 5-minute rule by May 24. It then proposes to issue a “draft determination” in July, 2017, and a final determination in September, which means it could be late 2020 before the changes are actually put in place.