Re: Chromium updater question

688 views
Skip to first unread message

Noah Rose Ledesma

unread,
Jul 13, 2023, 6:28:48 PM7/13/23
to Paweł, chrome-up...@chromium.org, gan...@chromium.org



On Thu, Jul 13, 2023 at 11:20 AM Paweł <pa...@gener8ads.com> wrote:
Hello Chromiumers,

I reach out to you as your names are mentioned as the first in the OWNERS file for the updater code. I hope you do not mind me contacting you directly. If this is not the best way please point me at such.
I'd like to switch to chromium updater in a chromium fork I work  on (win and mac)  and it works like a charm on windows and nicely replaces omaha updater. However I'm struggling to get it running on macOS. It seems the browser properly installs the updater and launches it as a local server but as soon as the mojo is connected both ends disconnect and so proxy calls to e.g. RegisterApp() or Update() timeout without response as they do not even reach the stub. I see - the same based on the log - when running chromium (google chrome to be more precise for that matter) with the updater flag enabled. It looks like something in mojo (the server does not exit prematurely and I see no obvious reason why the communication does not work) but it's hard to believe such a central element would misbehave here (and I debugged it for the past 2 days making circles and little progress). The question I have is: is this something known? Can you help me with this, point to places to look at, please? Moreover, what's the rollout plan for it? I see neither Chrome nor Edge use it and the rollout plan IIRC from the documentation mentioned rolling it out from early 2022.
Thanks in advance,

BR,
Pawel, chromium fan

Joshua Pawlicki

unread,
Jul 14, 2023, 12:18:26 PM7/14/23
to Noah Rose Ledesma, Paweł, chrome-up...@chromium.org, gan...@chromium.org
> as soon as the mojo is connected both ends disconnect
That's surprising. Does updater.log say anything? Which version of the browser and which version of the updater are you running? Is the updater operating in system or user scope? (I would guess per-user.)

>  is this something known?
No, I don't think so. Are you running with the kUseChromiumUpdater flag enabled? We are not yet doing that in production, and it's possible there's a bug in browser_updater_client_util_mac.mm's EnsureUpdater.

> Moreover, what's the rollout plan for it?
I can't speak for Edge but on the Chrome side, we've rolled out to a small percentage of the population and hope to expand to 100% over the next couple months.

--
You received this message because you are subscribed to the Google Groups "Chrome Updates Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-updates-...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-updates-dev/CAL%2BQ5O4UP9NnZt9NWPboK6vBq61_%3DOQT24XwT%3DNqcWJRxHyCkQ%40mail.gmail.com.

Paweł

unread,
Jul 14, 2023, 3:54:41 PM7/14/23
to Joshua Pawlicki, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org
Hi Joshua,

Thanks for coming back to me.
The chromium I work with is 115.0.5790.78 (the updater has the same version as I build it from the same sources and do not bundle any version with the browser). It operates in user scope, yes.
Google chrome OTOH is 115.0.5790.78 and the updater is 117.0.5860.0.
Unfortunately the log does not say much - unless I'm looking at it wrong. These are the snippets from running Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --enable-features="UseChromiumUpdater" --enable-logging=stderr --vmodule="*update*=9,*ipc*=9,*/*update*/*=9" (In the chromium I use I flipped the flag to be enabled by default):

The Browser side:
[78709:259:0714/161904.322661:VERBOSE1:update_service_proxy_posix.cc(250)] GetAppStates
[78709:259:0714/161904.322719:VERBOSE1:update_service_proxy_posix.cc(417)] EnsureConnecting
[78709:33031:0714/161904.323035:VERBOSE1:named_mojo_ipc_server_client_util.cc(43)] mach_msg_send: (ipc/send) invalid destination port (0x10000003)
[78709:33031:0714/161904.333957:VERBOSE1:named_mojo_ipc_server_client_util.cc(43)] mach_msg_send: (ipc/send) invalid destination port (0x10000003)
[78709:33031:0714/161904.369843:VERBOSE1:named_mojo_ipc_server_client_util.cc(43)] mach_msg_send: (ipc/send) invalid destination port (0x10000003)
[78709:33031:0714/161904.433917:VERBOSE1:named_mojo_ipc_server_client_util.cc(43)] mach_msg_send: (ipc/send) invalid destination port (0x10000003)
[78709:33031:0714/161904.532846:VERBOSE1:named_mojo_ipc_server_client_util.cc(43)] mach_msg_send: (ipc/send) invalid destination port (0x10000003)
[78709:259:0714/161904.657802:VERBOSE1:update_service_proxy_posix.cc(376)] OnConnected
[78709:259:0714/161904.665432:VERBOSE1:update_service_proxy_posix.cc(405)] OnDisconnected
[78709:259:0714/161904.665591:VERBOSE1:update_service_proxy_posix.cc(201)] GetVersion
[78709:259:0714/161904.665614:VERBOSE1:update_service_proxy_posix.cc(417)] EnsureConnecting
[78709:259:0714/161904.665782:VERBOSE1:update_service_proxy_posix.cc(376)] OnConnected
[78709:259:0714/161904.666193:VERBOSE1:update_service_proxy_posix.cc(405)] OnDisconnected
[78709:259:0714/161904.666272:VERBOSE1:browser_updater_client.cc(68)] Detected updater version: invalid
[78709:259:0714/161904.666304:VERBOSE1:update_service_proxy_posix.cc(412)] ~UpdateServiceProxy
[78920:259:0714/161904.737851:VERBOSE1:util.cc(279)] Log initialized for /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/114.0.5735.198/Helpers/GoogleUpdater.app/Contents/MacOS/GoogleUpdater -> /Users/gener8/Library/Application Support/Google/GoogleUpdater/updater.log
[78920:259:0714/161904.738298:VERBOSE1:updater.cc(275)] Version 114.0.5694.0, debug, 64 bits, command line: /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/114.0.5735.198/Helpers/GoogleUpdater.app/Contents/MacOS/GoogleUpdater --crash-handler --enable-logging --vmodule=*/components/update_client/*=2,*/chrome/updater/*=2 --database=/Users/gener8/Library/Application Support/Google/GoogleUpdater/114.0.5694.0/Crashpad --url=https://clients2.google.com/cr/report --annotation=prod=Update4 --annotation=ver=114.0.5694.0 --handshake-fd=5
[78920:259:0714/161904.804161:VERBOSE1:util.cc(279)] Log initialized for /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/114.0.5735.198/Helpers/GoogleUpdater.app/Contents/MacOS/GoogleUpdater -> /Users/gener8/Library/Application Support/Google/GoogleUpdater/updater.log
[78920:259:0714/161904.804205:VERBOSE1:updater.cc(278)] UpdaterMain (--crash-handler) returned 0.
[78709:259:0714/161904.832841:VERBOSE1:update_service_proxy_posix.cc(240)] RegisterApp
[78709:259:0714/161904.832879:VERBOSE1:update_service_proxy_posix.cc(417)] EnsureConnecting
[78709:259:0714/161904.833090:VERBOSE1:update_service_proxy_posix.cc(376)] OnConnected
[78709:259:0714/161904.833307:VERBOSE1:update_service_proxy_posix.cc(405)] OnDisconnected
[78709:259:0714/161904.833923:VERBOSE1:browser_updater_client.cc(51)] Updater registration error: 35
[78709:259:0714/161904.833966:VERBOSE1:update_service_proxy_posix.cc(412)] ~UpdateServiceProxy
[78709:259:0714/161904.834060:VERBOSE1:update_service_proxy_posix.cc(301)] Update
[78709:259:0714/161904.834088:VERBOSE1:update_service_proxy_posix.cc(417)] EnsureConnecting
[78709:259:0714/161904.834360:VERBOSE1:update_service_proxy_posix.cc(376)] OnConnected
[78709:259:0714/161904.834609:VERBOSE1:update_service_proxy_posix.cc(405)] OnDisconnected
[78709:259:0714/161904.834718:VERBOSE1:browser_updater_client.cc(93)] Result of update was: 9
[78709:259:0714/161904.834769:VERBOSE1:update_service_proxy_posix.cc(412)] ~UpdateServiceProxy

cat ~/Library/Application\ Support/Google/GoogleUpdater/updater.log:

[78909:259:0714/161904.399876:VERBOSE1:util.cc(268)] Log initialized for /Users/gener8/Library/Application Support/Google/GoogleUpdater/117.0.5860.0/GoogleUpdater.app/Contents/MacOS/GoogleUpdater -> /Users/gener8/Library/Application Support/Google/GoogleUpdater/updater.log
[78909:259:0714/161904.402431:VERBOSE1:updater.cc(278)] Version 117.0.5860.0, opt, 64 bits, command line: GoogleUpdater --server --service=update --enable-logging --vmodule=*/components/update_client/*=2,*/chrome/updater/*=2
[78911:259:0714/161904.415638:VERBOSE1:util.cc(268)] Log initialized for /Users/gener8/Library/Application Support/Google/GoogleUpdater/117.0.5860.0/GoogleUpdater.app/Contents/MacOS/GoogleUpdater -> /Users/gener8/Library/Application Support/Google/GoogleUpdater/updater.log
[78911:259:0714/161904.415986:VERBOSE1:updater.cc(278)] Version 117.0.5860.0, opt, 64 bits, command line: /Users/gener8/Library/Application Support/Google/GoogleUpdater/117.0.5860.0/GoogleUpdater.app/Contents/MacOS/GoogleUpdater --crash-handler --enable-logging --vmodule=*/components/update_client/*=2,*/chrome/updater/*=2 --database=/Users/gener8/Library/Application Support/Google/GoogleUpdater/117.0.5860.0/Crashpad --url=https://clients2.google.com/cr/report --annotation=prod=Update4 --annotation=ver=117.0.5860.0 --handshake-fd=5
[78909:259:0714/161904.418334:VERBOSE1:crash_reporter.cc(110)] Crash handler launched and ready.
[78909:259:0714/161904.421624:VERBOSE1:crash_client.cc(100)] Found 0 completed crash reports
[78909:259:0714/161904.421740:VERBOSE1:crash_client.cc(118)] Found 0 pending crash reports
[78909:259:0714/161904.428443:VERBOSE1:updater.cc(91)] Crash reporting initialized.
[78909:259:0714/161904.500507:VERBOSE2:app_server.cc(55)] ModeCheck
[78909:259:0714/161904.500890:VERBOSE2:lock_mac.mm(48)] TryAcquireReceive
[78909:259:0714/161904.501178:VERBOSE1:prefs.cc(162)] global_prefs_dir: /Users/gener8/Library/Application Support/Google/GoogleUpdater
[78909:259:0714/161904.503186:VERBOSE1:prefs.cc(92)] UpdaterPrefsImpl
[78909:259:0714/161904.503241:VERBOSE2:app_server.cc(68)] This version: 117.0.5860.0, active version: 117.0.5860.0
[78909:259:0714/161904.506602:VERBOSE1:service.cc(113)] Current effective policies:
{
  LastCheckPeriod = 270 (default)
}

[78909:259:0714/161904.665889:VERBOSE1:update_service_stub.cc(409)] UpdateService client disconnected.
[78909:259:0714/161904.666208:VERBOSE1:update_service_stub.cc(409)] UpdateService client disconnected.
[78920:259:0714/161904.737851:VERBOSE1:util.cc(279)] Log initialized for /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/114.0.5735.198/Helpers/GoogleUpdater.app/Contents/MacOS/GoogleUpdater -> /Users/gener8/Library/Application Support/Google/GoogleUpdater/updater.log
[78920:259:0714/161904.738298:VERBOSE1:updater.cc(275)] Version 114.0.5694.0, debug, 64 bits, command line: /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/114.0.5735.198/Helpers/GoogleUpdater.app/Contents/MacOS/GoogleUpdater --crash-handler --enable-logging --vmodule=*/components/update_client/*=2,*/chrome/updater/*=2 --database=/Users/gener8/Library/Application Support/Google/GoogleUpdater/114.0.5694.0/Crashpad --url=https://clients2.google.com/cr/report --annotation=prod=Update4 --annotation=ver=114.0.5694.0 --handshake-fd=5
[78909:259:0714/161904.801407:VERBOSE1:update_service_impl.cc(233)] GetVersion
[78909:259:0714/161904.802583:VERBOSE1:update_service_stub.cc(409)] UpdateService client disconnected.
[78920:259:0714/161904.804161:VERBOSE1:util.cc(279)] Log initialized for /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/114.0.5735.198/Helpers/GoogleUpdater.app/Contents/MacOS/GoogleUpdater -> /Users/gener8/Library/Application Support/Google/GoogleUpdater/updater.log
[78920:259:0714/161904.804205:VERBOSE1:updater.cc(278)] UpdaterMain (--crash-handler) returned 0.
[78909:259:0714/161904.833328:VERBOSE1:update_service_stub.cc(409)] UpdateService client disconnected.
[78909:259:0714/161904.834628:VERBOSE1:update_service_stub.cc(409)] UpdateService client disconnected.
[78909:26119:0714/161914.801351:VERBOSE1:prefs.cc(209)] Prefs committed.
[78909:259:0714/161914.801461:VERBOSE1:prefs.cc(96)] ~UpdaterPrefsImpl
[78909:259:0714/161914.801715:VERBOSE1:updater.cc(281)] UpdaterMain (--server) returned 0.
[78911:259:0714/161914.803556:VERBOSE1:util.cc(268)] Log initialized for /Users/gener8/Library/Application Support/Google/GoogleUpdater/117.0.5860.0/GoogleUpdater.app/Contents/MacOS/GoogleUpdater -> /Users/gener8/Library/Application Support/Google/GoogleUpdater/updater.log
[78911:259:0714/161914.803882:VERBOSE1:updater.cc(281)] UpdaterMain (--crash-handler) returned 0.

I gathered these by loading chrome://settings/help, clearing both the console and the updater.log and reloading.
The updater lands under  ~/Library/Application\ Support/Google/GoogleUpdater/ so I'd assume that rules out a bug in EnsureUpdater() (114.0.5694.0 is also there for me)?

Paweł

unread,
Jul 14, 2023, 4:00:17 PM7/14/23
to Joshua Pawlicki, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org
Correction to the above:
Chrome is ofc in the latest stable version (I spotted I gave the version of chromium I use twice) i.e. 114.0.5694.198.
Also I gathered the log like this just to limit it to minimum because on start a lot is going on wrt update e.g. components are checked for update in the browser.

Joshua Pawlicki

unread,
Jul 20, 2023, 10:37:38 PM7/20/23
to Paweł, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org
Apologies for the delay, I was out of office (and then catching up).

Thanks, these logs help. I suspect --enable-features="UseChromiumUpdater" is simply broken in M114 (and maybe even in M115 or later), due to changes we've made in the proxy dialing mechanisms. We haven't been using it. (Chrome should still be able to communicate with GoogleUpdater without the feature flag set, via the Keystone compatibility shims.) Still, it might be worth digging into this a little more.

Is /Applications/Google Chrome.app owned by gener8 or by root?

Can you reproduce the issue on M117? (e.g. a Chrome Canary install?)

I'll also take some time tomorrow to repro locally.

Paweł

unread,
Jul 21, 2023, 9:11:30 AM7/21/23
to Joshua Pawlicki, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org, roc...@google.com
Hi Joshua,

I had to switch to something else from Monday this week. I however did narrow it down to mojo. I haven't found the exact issue but it's something with ipcz (that is now on by default in the browser yet forced to be off in the updater - see https://bugs.chromium.org/p/chromium/issues/detail?id=1418740). Disabling the flag on the browser end makes it all work. (I haven't tried to keep it on in the browser and turn it on in the updater).

fyi @rockot

Joshua Pawlicki

unread,
Jul 21, 2023, 11:30:44 AM7/21/23
to Paweł, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org, roc...@google.com
Ah ha! Yes, ipcz could be the reason. It looks like crbug.com/1418740 is unblocked as of 116.0.5817.0, and we should try re-enabling ipcz in the updater.

Ken Rockot

unread,
Jul 21, 2023, 5:24:42 PM7/21/23
to Joshua Pawlicki, Paweł, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org
Ahh hm, I didn't realize the browser and the updater communicated over Mojo. How does that work in Chrome today with ipcz shipping only in the browser? Or is this strictly a downstream use case?

Joshua Pawlicki

unread,
Jul 21, 2023, 5:31:08 PM7/21/23
to Ken Rockot, Paweł, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org
Today we fall back to communicating using COM / command line / stdout in Chrome, since we're not shipping Chrome with --enable-features="UseChromiumUpdater", but rather rolling out ChromiumUpdater through self-updates to Omaha and Keystone.

So it would be for downstreamers, for now.

At some point we'll flip that feature, though, but it sounds like we first need ipcz working in the updater again; and I guess we need to think carefully about how pre-ipcz ChromiumUpdaters will communicate with post-ipcz updaters.

Ken Rockot

unread,
Jul 21, 2023, 5:42:21 PM7/21/23
to Joshua Pawlicki, Paweł, Noah Rose Ledesma, chrome-up...@chromium.org, gan...@chromium.org
On Fri, Jul 21, 2023 at 10:31 AM Joshua Pawlicki <waf...@google.com> wrote:
Today we fall back to communicating using COM / command line / stdout in Chrome, since we're not shipping Chrome with --enable-features="UseChromiumUpdater", but rather rolling out ChromiumUpdater through self-updates to Omaha and Keystone.

So it would be for downstreamers, for now.

At some point we'll flip that feature, though, but it sounds like we first need ipcz working in the updater again; and I guess we need to think carefully about how pre-ipcz ChromiumUpdaters will communicate with post-ipcz updaters.

Hopefully we can do something simpler, but if at some point we need to smash the two worlds together, there's a useful (Chrome OS) prototype of a proxying service here. Shouldn't be hard to adapt to other platforms.
Reply all
Reply to author
Forward
0 new messages