From that, is seems like we should throw out the illusion of competition in our market and go back to either a government controlled monopoly or a heavily regulated commercial monopoly that has the required resources and the capacity to think in the long term ?
Putting some detail around the "why" aspect of that, there's three basic issues with the present market relating to the competition aspect which explain why it's not leading to the lower prices economists would argue it ought to.
1. Generation dispatch outcomes.
In layman's terms that's which power stations are running at any given time bearing in mind that, unless demand is at maximum, they won't all be needed. Eg demand in Victoria peaks at about 10.4GW but right now it's just 4.4GW so there's simply no need to run all the generation.
Some forms of generation are virtually free to operate once built and are "use it or lose it". Eg solar.
Some are almost free to operate but are volume limited over time. Eg hydro. There's only so much water entering the system, total production over time is constrained.
Some are not free but they're medium price to operate. Eg coal or the more efficient gas plants.
Some are massively expensive to operate, eg diesel and the less efficient gas plants.
Now from that it ought be obvious that the different forms of generation aren't really competing. It's never rational to run the diesel in preference to anything else that's sitting there unused but able to run. It's never rational to run anything in preference to using the solar.
When to run the hydro depends how much water's available but as a concept the idea is to run it instead of diesel first, then gas, then coal. You don't sensibly run it instead of the solar, drain the lake, then end up burning diesel since that would be silly.
So the different forms of generation aren't competing in any rational world but the NEM design insists that they do in fact compete and provides no option to do otherwise. End result is we do in fact end up with diesel being run and coal turned down. We do in fact end up turning solar off and burning gas. And so on. Serious $ wasted there, that's the first major flaw in the market.
2. Second major flaw is that retail costs far more than it really ought to.
Fundamentally it's reading meters, sending out bills and making sure they're paid. That's what a consumer needs a retailer to do.
However.....
Since we have a highly volatile market with pricing set every 5 minutes, whereas consumers prefer prices that are more stable, retailers hedge their market exposure via assorted hedging contracts.
So we have big $ spent on hedging contracts to hedge against the risk of price moving. Bearing in mind that the price movement is, of course, simply a product of what the traders do, it's not some fundamental thing driven by crop harvests or something as is the case with agriculture.
So we have manufactured volatility and a hefty price charged to hedge against that volatility. It's the equivalent of insurers going around smashing windows just to make sure you buy their rather expensive insurance.
Now add in that every retailer has a management, overheads and so on plus the big one, they need billing systems which comply with all the rules, they need procedures to facilitate efficient transfer of customers between retailers and so on. End result is a substantial portion of the customer's bill has nothing at all to do with any physical aspect of generating, transmitting or distributing electricity, being simply the cost to administer the whole thing and send the bills out.
Now that cost is, in % terms, higher than the total administrative budget of one of the state electricity authorities in 1962. Yes I've checked the figures. So back in the days of having to manually type out each invoice, having physical shopfronts taking cash payments and so on it was actually cheaper.
Competition can lower prices yes, but not when the means of having that competition is itself costing a fortune.
3. Scale of economy in generation.
Electricity generation is a somewhat extreme example of scale of economy, driven by the reality that operating a machine 10 times the size doesn't need 10 times as many people or even remotely close to it.
Go back 70 years and the largest steam turbines were 50MW. By about 1967 there were 500MW machines going into service. Today GE has a standard "off the shelf" 1900MW design available to be manufactured for anyone who wants to buy one.
So if we look at the nuclear plants being built overseas, practically all use units 1000MW or above for that reason, scale of economy. In the UK they're up around 1600MW. That's per machine not for the entire power station.
Same in the UAE. 4 x 1400MW at a single site. Done because that gets the cost down per unit of production.
It's fundamentally cheaper to have a few large power stations than to have lots of little ones. Hence (for examples) why WA ended up with most generation focused at Muja and Kwinana, why SA did it with Port Augusta and Torrens Island and why Victoria got the vast majority of supply from the Yallourn, Hazelwood (now gone) and Loy Yang complexes.
Large scale is the means of getting the cost down. Hence Victoria went from 50MW machines at Yallourn D, to 120MW in the next station Yallourn E, to 200MW at Hazelwood, to 350MW at Yallourn W to 500MW at Loy Yang. All about lowering costs.
For hydro natural topography does limit the scale of any individual station but the same basic concept applies, scale wins. Hence if we look at Tasmania as an example, the approach was unavoidably to build a large number of individually small power stations but they were very intentionally built in groups all at once. Eg the 7 Mersey-Forth stations were all built as a single project, they weren't built one at a time. Same with the others, they were grouped and done together so far as practical in order to economise on construction costs. There's still that benefit of scale to an extent.
Same with Snowy 2.0, it's far cheaper to store 350GWh at one site than to build 175 little 2GWh sites. The latter is technically viable sure, it's just a lot more expensive to have 175 tunnels, 175 power stations, 175 access roads and so on versus one big tunnel, one big power station, etc.
One consequence of that, of course, is it limits the number of owners. If you're only going to have a small number of individually large facilities then you simply can't have more than a few owners, and the owners you do have need to be financially and physically capable of taking on such huge projects.
The idea that we ought have a vast number of individually small companies competing against each other is totally at odds with that fundamental aspect of scale of economy. Add that to the issues with dispatch and retail and the result is prices going up not down despite competition, in theory, driving innovation and so on.
Where competition would work is with competition to decide what gets built in the first place, who builds and who owns. Have competing projects tendering, competing companies offering to own them at whatever cost of finance, and so on. In that case the economic fundamentals of competition aren't being undone by introduced technical inefficiencies.