I was playing around more and actually found what is probably the issue. UI Scaling in preferences. If I set it to the smallest panels work correctly. I had it on one up from smallest and that is where the problem is, and happens with all larger options in UI scaling. Been there for many versions. I am at least happy that I have a work around and can live with the smaller UI in hopes that eventually Adobe will fix this glitch.

This bug has been in Indesign a while now. using Mac OS Sonoma latest updates. This occurred in previous Mac OS's as well. Computer is M1 Max - MacBook Pro. This happens when using an external display, so 2 screens (1 MacBook Pro screen and 2nd via HDMI). Indesign is open on the larger secondary screen. If "Application Frame" is unchecked, opening tool bars snapped to the right side of the screen all open their windows right of the tool bar and thus go off screen. They should open to the left.

This video shows it working correctly with Application Frame selected. This is on the secondary screen, panels on the right are what I am clicking:
https://gyazo.com/125bfea03128570c194acf19973138da

This is the same thing but with Application Frame turned off. Tool panels are snapped to the right of the screen. When clicked on the windows expand and open to the right of the panels rather then how they should on the left.
https://gyazo.com/0d9b624d0152c035913535294b5cb1cf

This only happens when an external display is connected. If I quit Indesign, unplug the secondary screen and work off the MacBook Pro screen the panels open correctly with Application Frame turned off.

Same! What also works if you drag the panel out of the stack.

I am also getting this problem. When opening a document I have to close the windows down so I can see the doc and the tool windows at the same time. I was previously using v16.0.2 and it doesn't have this problem. I have tried the previous commenter suggestion, about deleting the. Library Caches for Indesign and it did not work to fix the problem

Fixed! I had to change the UI scaling to the smallest setting. After that change and a relaunch, the tool palettes expanded properly

Not an "idea" but a huge bug. In Design 2023 (v18.0) just updated today. Since update, no tool palettes expand. The step to reproduce my issue is to try and use any palette. The expected result was for the palettes to operate normally, like prior to the update. The actual result was a stack of 100% useless tool palettes.

For macOS, this method may correct the problem.
1. Quit InDesign.
2. Go to User > Library > Caches.
3. Search for “indesign” and select and delete all of them if detected.

The side panels will not open unless the application frame is on. It has been this way for over a year now and no updates have solved it. I am sick of having to have the toolbar floating instead of fixed to the side. Have just updated to InDesign 2023 and still a problem. And yes, I have deleted / reset my preferences several times which has done nothing.

I just updated to 17.4 and I'm experiencing the same issue. The panels are not working correctly if you aren't in Application Frame mode.

This problem has happened before and now its back in version 17.4 you click custom panels and nothing happens, you can only switch them in 'Application Frame' mode

I have the same problem on my iMac (late 2015), when I turn off Application Frame, panels won't open from icons, I need to keep all panels expanded.
This is very annoying for me.

I generally do not have my application frame turned on. I know this is a PC thing, but just not natural on a Mac. Often if I donʻt have application frame turned on, many of the panels like layers simply wonʻt display no matter what I do. This seems like a bug to me.

Also there is a bug where the item dot in a layer is not visible so you canʻt move an item easily to another layer. You have to rebuild preferences to get the dot to show again. Again another bug Adobe should be fixing before addin new features. Adobeʻs attention to the Mac just doesnʻt seem to be there. They seemed to be completely PC focused. Mac apps could be so much better, faster, more efficient (Adobe premiere big example), but they hold themselves back to satisfy PC market. Dissapointing since they really got their start on the Mac.