Google mail (workspace) is annoying but most clients are using it so we need it to work on moqui. The problem is google is phasing out less secure apps. I didn’t research much but I think we need oauth for email access or something like that. Did anyone solve this problem? What’s the best way to integrate moqui with google mail?
1 Like
We partly added OAUTH support (polling side) for office365; it should be similar for Gmail. It could be cleaned up and added into moqui. We did not implement the sending side…it just turned out to be a lot easier to use Amazon SES, so we stopped working on it.
We extended moqui.basic.email.EmailServer, adding:
<!--
Office365: https://www.flowable.com/blog/engineering/office-365-exchange-oauth2-imap-authentication
Authentication tokens may be decoded here: https://jwt.ms/
The sub on the token represents the authEnterpriseSubObjectId, but ensure it has been added as a principal (via powershell)
to link the enterprise version of the application.
The following will need run via PowerShell for Office 365:
New-ServicePrincipal -AppId "authAppClientId" -ServiceId "authEnterpriseSubObjectId"
Set-ServicePrincipal -Identity "authEnterpriseSubObjectId" -DisplayName "pr"
Add-MailboxPermission -Identity "email@address.com" -User "authEnterpriseSubObjectId" -AccessRights FullAccess
If the above permissions have not been added:
https://learn.microsoft.com/en-us/answers/questions/1195069/javax-mail-authenticationfailedexception-authentic
-->
<field name="authMechanisms" type="text-medium"><description>e.g. XOAUTH2</description></field>
<!-- Login URL composed as: loginPost authLoginUrl . authTenantId . authLoginUrlSuffix -->
<field name="authLoginUrl" type="text-medium"><description>e.g. https://login.microsoftonline.com/</description></field>
<field name="authLoginUrlSuffix" type="text-medium"><description>e.g. /oauth2/v2.0/token</description></field>
<field name="authScopesUrl" type="text-medium"><description>e.g. https://outlook.office365.com/.default</description></field>
<field name="authAppClientId" type="text-medium" encrypt="true"/>
<field name="authObjectId" type="text-medium" encrypt="true"/>
<field name="authTenantId" type="text-medium" encrypt="true"/>
<field name="authClientSecretId" type="text-medium" encrypt="true"/>
<field name="authClientSecretValue" type="text-medium" encrypt="true"/>
<field name="authEnterpriseSubObjectId" type="text-medium" encrypt="true">
<description>Ensure IMAP.AccessAsUser.All has been added to the application.</description>
</field>
We are still on Moqui 2 and have some Java 8 clients: this is our pollEmailServer.groovy:
/*
* This software is in the public domain under CC0 1.0 Universal plus a
* Grant of Patent License.
*
* To the extent possible under law, the author(s) have dedicated all
* copyright and related and neighboring rights to this software to the
* public domain worldwide. This software is distributed without any
* warranty.
*
* You should have received a copy of the CC0 Public Domain Dedication
* along with this software (see the LICENSE.md file). If not, see
* <http://creativecommons.org/publicdomain/zero/1.0/>.
*/
/*
JavaMail API Documentation at: https://java.net/projects/javamail/pages/Home
For JavaMail JavaDocs see: https://javamail.java.net/nonav/docs/api/index.html
*/
import javax.mail.FetchProfile
import javax.mail.Flags
import javax.mail.Folder
import javax.mail.Message
import javax.mail.Session
import javax.mail.Store
import javax.mail.internet.MimeMessage
import javax.mail.search.FlagTerm
import javax.mail.search.SearchTerm
import org.apache.http.client.ClientProtocolException
import org.apache.http.impl.client.CloseableHttpClient
import org.apache.http.client.methods.HttpPost
import org.apache.http.client.methods.CloseableHttpResponse
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.JavaType
import org.apache.http.message.BasicHeader
import org.apache.http.impl.client.HttpClients
import org.apache.http.entity.StringEntity
import org.apache.http.entity.ContentType
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import org.moqui.entity.EntityValue
import org.moqui.impl.context.ExecutionContextImpl
Logger logger = LoggerFactory.getLogger("org.moqui.impl.pollEmailServer")
// This is provided for backwards compatibility with Java 1.8
public static byte[] readAllBytes(InputStream inputStream) throws IOException {
final int bufLen = 1024
byte[] buf = new byte[bufLen]
int readLen
IOException exception = null
try {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream()
while ((readLen = inputStream.read(buf, 0, bufLen)) != -1)
outputStream.write(buf, 0, readLen)
return outputStream.toByteArray()
} catch (IOException e) {
exception = e
throw e
} finally {
// TODO: This closes the inputStream; should it?
if (exception == null) inputStream.close()
else try {
inputStream.close()
} catch (IOException e) {
exception.addSuppressed(e)
}
}
}
// TODO: This may be Office365 specific despite being parameterized
public String getAuthToken(String authTenantId,
String authAppClientId,
String authClientSecretValue,
String authLoginUrl,
String authLoginUrlSuffix,
String authScopesUrl) throws ClientProtocolException, IOException {
CloseableHttpClient client = HttpClients.createDefault()
// HttpPost loginPost = new HttpPost("https://login.microsoftonline.com/" + authTenantId + "/oauth2/v2.0/token")
HttpPost loginPost = new HttpPost(authLoginUrl + authTenantId + authLoginUrlSuffix)
// String scopes = "https://outlook.office365.com/.default"
String scopes = authScopesUrl
String encodedBody = "client_id=" + authAppClientId + "&scope=" + scopes + "&client_secret=" + authClientSecretValue + "&grant_type=client_credentials"
loginPost.setEntity(new StringEntity(encodedBody, ContentType.APPLICATION_FORM_URLENCODED))
loginPost.addHeader(new BasicHeader("cache-control", "no-cache"))
CloseableHttpResponse loginResponse = client.execute(loginPost)
InputStream inputStream = loginResponse.getEntity().getContent()
byte[] response = readAllBytes(inputStream) // Call Java 1.8 compatible readAllBytes
ObjectMapper objectMapper = new ObjectMapper()
JavaType type = objectMapper.constructType(
objectMapper.getTypeFactory().constructParametricType(Map.class, String.class, String.class))
Map<String, String> parsed = new ObjectMapper().readValue(response, type)
return parsed.get("access_token")
}
ExecutionContextImpl ec = context.ec
EntityValue emailServer = ec.entity.find("moqui.basic.email.EmailServer").condition("emailServerId", emailServerId).one()
if (!emailServer) { ec.message.addError(ec.resource.expand('No EmailServer found for ID [${emailServerId}]','')); return }
if (!emailServer.storeHost) { ec.message.addError(ec.resource.expand('EmailServer [${emailServerId}] has no storeHost','')) }
if (!emailServer.mailUsername) { ec.message.addError(ec.resource.expand('EmailServer [${emailServerId}] has no mailUsername','')) }
if (!emailServer.mailPassword && !emailServer.authMechanisms) { ec.message.addError(ec.resource.expand('EmailServer [${emailServerId}] has no mailPassword and no authMechanisms','')) }
if (ec.message.hasError()) return
String host = emailServer.storeHost
String user = emailServer.mailUsername
String password = emailServer.mailPassword
String protocol = emailServer.storeProtocol ?: "imaps"
int port = (emailServer.storePort ?: "993") as int
String storeFolder = emailServer.storeFolder ?: "INBOX"
// def urlName = new URLName(protocol, host, port as int, "", user, password)
Session session
Store store
if (emailServer.authMechanisms) {
String authMechanisms = emailServer.authMechanisms
String authLoginUrl = emailServer.authLoginUrl
String authLoginUrlSuffix = emailServer.authLoginUrlSuffix
String authScopesUrl = emailServer.authScopesUrl
String authAppClientId = emailServer.authAppClientId
// String authObjectId = emailServer.authObjectId
String authTenantId = emailServer.authTenantId
// String authClientSecretId = emailServer.authClientSecretId
String authClientSecretValue = emailServer.authClientSecretValue
// String authEnterpriseSubObjectId = emailServer.authEnterpriseSubObjectId
Properties props = new Properties()
// TODO: This is IMAP specific
props.put("mail.store.protocol", protocol) // Some documentation suggests this should be imap, not imaps
props.put("mail.imap.host", host)
props.put("mail.imap.port", port)
// props.put("mail.imap.ssl.enable", "true")
props.put("mail.imap.ssl.enable", emailServer.smtpSsl == 'N' ? "false": "true")
// props.put("mail.imap.starttls.enable", "true")
props.put("mail.imap.starttls.enable", emailServer.smtpStartTls == 'N' ? "false": "true")
props.put("mail.imap.auth", "true")
props.put("mail.imap.auth.mechanisms", authMechanisms)
props.put("mail.imap.user", user)
// props.put("mail.debug", "true")
// props.put("mail.debug.auth", "true")
String token = getAuthToken(authTenantId,
authAppClientId,
authClientSecretValue,
authLoginUrl,
authLoginUrlSuffix,
authScopesUrl)
// logger.warn('token: ' + token.toString())
session = Session.getInstance(props)
// session.setDebug(true)
store = session.getStore("imap")
store.connect("outlook.office365.com", user, token)
}
else {
session = Session.getInstance(System.getProperties())
store = session.getStore(protocol)
if (!store.isConnected()) store.connect(host, port, user, password)
}
logger.info("Polling Email from ${user}@${host}:${port}/${storeFolder}, properties ${session.getProperties()}")
// open the folder
Folder folder = store.getFolder(storeFolder)
if (folder == null || !folder.exists()) { ec.message.addError(ec.resource.expand('No ${storeFolder} folder found','')); return }
// get message count
folder.open(Folder.READ_WRITE)
int totalMessages = folder.getMessageCount()
// close and return if no messages
if (totalMessages == 0) { folder.close(false); return }
// get messages not deleted (and optionally not seen)
Flags searchFlags = new Flags(Flags.Flag.DELETED)
if (emailServer.storeSkipSeen == "Y") searchFlags.add(Flags.Flag.SEEN)
SearchTerm searchTerm = new FlagTerm(searchFlags, false)
Message[] messages = folder.search(searchTerm)
FetchProfile profile = new FetchProfile()
profile.add(FetchProfile.Item.ENVELOPE)
profile.add(FetchProfile.Item.FLAGS)
profile.add("X-Mailer")
folder.fetch(messages, profile)
logger.info("Found ${totalMessages} messages (${messages.size()} filtered) at ${user}@${host}:${port}/${storeFolder}")
for (Message message in messages) {
if (emailServer.storeSkipSeen == "Y" && message.isSet(Flags.Flag.SEEN)) continue
// NOTE: should we check size? long messageSize = message.getSize()
if (message instanceof MimeMessage) {
// use copy constructor to have it download the full message, may fix BODYSTRUCTURE issue from some email servers (see details in issue #97)
MimeMessage fullMessage = new MimeMessage(message)
ec.service.runEmecaRules(fullMessage, emailServerId)
// mark seen if setup to do so
if (emailServer.storeMarkSeen == "Y") message.setFlag(Flags.Flag.SEEN, true)
// delete the message if setup to do so
if (emailServer.storeDelete == "Y") message.setFlag(Flags.Flag.DELETED, true)
} else {
logger.warn("Doing nothing with non-MimeMessage message: ${message}")
}
}
// expunge and close the folder
folder.close(true)
2 Likes
Also, by “Stopped working on it”… that doesn’t mean it doesn’t work…this is in production and running against an office365 email account.
It just means we didn’t take it further to clean it up or add the sending side of things.
1 Like
This depends on what you’re using gmail for.
I’m assuming you’re using it for sending emails from an email address per client that they configure.
It looks like you could go down a couple routes for this.
- Email over OAuth2: Despite google phasing out less secure apps, they google has modified their imap, pop, and smtp servers to use OAuth authentication using xoauth2-protocol. Google recommends to use the javamail library which has built in OAuth2 as specified by google. Moqui is currently using Apache Commons mail which does not support OAuth2 authentication. The oauth2 mechanism for this could be done well with a bit of effort assuming the libraries behave. This could even be pushed upstream for supporting email over OAuth2.
- The alternative option is Email over REST API. Depending on the requirements of the application, if only a small amount of the APIs are needed, then google would prefer this approach to limit the scope of the application (you can find scopes here). You should be able to autheticate with OAuth2 using the moqui-sso component. After authenticating with OAuth2 to the google account with your application, you would send a HTTP Post to the user.messages.send endpoint. This might be simpler overall, and setting up the infrastructure here could allow for Moqui calling other google api endpoints.
There’s some thoughts. Hopefully that helps!
1 Like