在实际使用Presto的过程中,经常会有以下的一些需求。

  • 添加一个新的Catalog
  • 对不再使用的Catalog希望把它删除
  • 修改某个Catalog的参数

但在Presto中如果进行上述的修改,需要重启Presto服务才可以生效,这给集群维护带来额外的工作量之外,还给上层应用带来很不好的使用体验。
如果还不能在开发环境很好运行Presto的话,参考在windows的IDEA运行Presto。 观察PrestoServer的run方法,可以知道Presot分了多个模块,而且多个模块依赖airlift(Airlift framework for building REST services)的项目,倒不如说airlift是Presto的根基。说实在话,我对于Presto的理解可能还在管中窥豹的阶段,但是不影响我对它做一些手脚。

1、新增一个Hello world测试接口

在presto-main中com.facebook.presto.server中新增一个CatalogResource,作用和Spring类似吧。

  1. /*
  2. * Licensed under the Apache License, Version 2.0 (the "License");
  3. * you may not use this file except in compliance with the License.
  4. * You may obtain a copy of the License at
  5. *
  6. * http://www.apache.org/licenses/LICENSE-2.0
  7. *
  8. * Unless required by applicable law or agreed to in writing, software
  9. * distributed under the License is distributed on an "AS IS" BASIS,
  10. * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  11. * See the License for the specific language governing permissions and
  12. * limitations under the License.
  13. */
  14. package com.facebook.presto.server;
  15. import javax.inject.Inject;
  16. import javax.ws.rs.GET;
  17. import javax.ws.rs.Path;
  18. import javax.ws.rs.core.Response;
  19. /**
  20. * @author Gin
  21. * @since 2018/8/15.
  22. */
  23. @Path("/v1/catalog")
  24. public class CatalogResource
  25. {
  26. @Inject
  27. public CatalogResource()
  28. {
  29. }
  30. @GET
  31. @Path("test")
  32. public Response test()
  33. {
  34. return Response.ok("Hello world").build();
  35. }
  36. }

在ServerMainModule的setup()的方法中最后一行注册CatalogResource:

  1. // Catalogs
  2. jaxrsBinder(binder).bind(CatalogResource.class);

启动server,访问http://localhost:8080/v1/catalog/test,如果出现HelloWorld的话,那么后面的步骤才行得通。

2、新增一个Add Connector的RESTful接口

新建CatalogInfo用于接收参数:

  1. package com.facebook.presto.server;
  2. import com.fasterxml.jackson.annotation.JsonCreator;
  3. import com.fasterxml.jackson.annotation.JsonProperty;
  4. import java.util.Map;
  5. import static java.util.Objects.requireNonNull;
  6. /**
  7. * @author Gin
  8. * @since 2018/8/17.
  9. */
  10. public class CatalogInfo
  11. {
  12. private final String catalogName;
  13. private final String connectorName;
  14. private final Map<String, String> properties;
  15. @JsonCreator
  16. public CatalogInfo(
  17. @JsonProperty("catalogName") String catalogName,
  18. @JsonProperty("connectorName") String connectorName,
  19. @JsonProperty("properties")Map<String, String> properties)
  20. {
  21. this.catalogName = requireNonNull(catalogName, "catalogName is null");
  22. this.connectorName = requireNonNull(connectorName, "connectorName is null");
  23. this.properties = requireNonNull(properties, "properties is null");
  24. }
  25. @JsonProperty
  26. public String getCatalogName() {
  27. return catalogName;
  28. }
  29. @JsonProperty
  30. public String getConnectorName() {
  31. return connectorName;
  32. }
  33. @JsonProperty
  34. public Map<String, String> getProperties() {
  35. return properties;
  36. }
  37. }

在CatalogResource中增加对应的接口,用到的服务用注入方法声明。

  1. /*
  2. * Licensed under the Apache License, Version 2.0 (the "License");
  3. * you may not use this file except in compliance with the License.
  4. * You may obtain a copy of the License at
  5. *
  6. * http://www.apache.org/licenses/LICENSE-2.0
  7. *
  8. * Unless required by applicable law or agreed to in writing, software
  9. * distributed under the License is distributed on an "AS IS" BASIS,
  10. * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  11. * See the License for the specific language governing permissions and
  12. * limitations under the License.
  13. */
  14. package com.facebook.presto.server;
  15. import com.facebook.presto.connector.ConnectorId;
  16. import com.facebook.presto.connector.ConnectorManager;
  17. import com.facebook.presto.metadata.InternalNodeManager;
  18. import com.google.common.base.Joiner;
  19. import com.google.common.base.Splitter;
  20. import io.airlift.discovery.client.Announcer;
  21. import io.airlift.discovery.client.ServiceAnnouncement;
  22. import javax.inject.Inject;
  23. import javax.ws.rs.*;
  24. import javax.ws.rs.core.MediaType;
  25. import javax.ws.rs.core.Response;
  26. import java.util.LinkedHashMap;
  27. import java.util.LinkedHashSet;
  28. import java.util.Map;
  29. import java.util.Set;
  30. import static com.google.common.base.Strings.nullToEmpty;
  31. import static io.airlift.discovery.client.ServiceAnnouncement.serviceAnnouncement;
  32. import static java.util.Objects.requireNonNull;
  33. /**
  34. * @author Gin
  35. * @since 2018/8/15.
  36. */
  37. @Path("/v1/catalog")
  38. public class CatalogResource
  39. {
  40. private final ConnectorManager connectorManager;
  41. private final Announcer announcer;
  42. @Inject
  43. public CatalogResource(
  44. ConnectorManager connectorManager,
  45. Announcer announcer)
  46. {
  47. this.connectorManager = requireNonNull(connectorManager, "connectorManager is null");
  48. this.announcer = requireNonNull(announcer, "announcer is null");
  49. }
  50. @GET
  51. @Path("test")
  52. public Response test()
  53. {
  54. return Response.ok("Hello world").build();
  55. }
  56. @PUT
  57. @Consumes(MediaType.APPLICATION_JSON)
  58. @Produces(MediaType.APPLICATION_JSON)
  59. public Response createCatalog(CatalogInfo catalogInfo)
  60. {
  61. requireNonNull(catalogInfo, "catalogInfo is null");
  62. ConnectorId connectorId = connectorManager.createConnection(
  63. catalogInfo.getCatalogName(),
  64. catalogInfo.getConnectorName(),
  65. catalogInfo.getProperties());
  66. updateConnectorIdAnnouncement(announcer, connectorId);
  67. return Response.status(Response.Status.OK).build();
  68. }
  69. private static void updateConnectorIdAnnouncement(Announcer announcer, ConnectorId connectorId)
  70. {
  71. //
  72. // This code was copied from PrestoServer, and is a hack that should be removed when the connectorId property is removed
  73. //
  74. // get existing announcement
  75. ServiceAnnouncement announcement = getPrestoAnnouncement(announcer.getServiceAnnouncements());
  76. // update connectorIds property
  77. Map<String, String> properties = new LinkedHashMap<>(announcement.getProperties());
  78. String property = nullToEmpty(properties.get("connectorIds"));
  79. Set<String> connectorIds = new LinkedHashSet<>(Splitter.on(',').trimResults().omitEmptyStrings().splitToList(property));
  80. connectorIds.add(connectorId.toString());
  81. properties.put("connectorIds", Joiner.on(',').join(connectorIds));
  82. // update announcement
  83. announcer.removeServiceAnnouncement(announcement.getId());
  84. announcer.addServiceAnnouncement(serviceAnnouncement(announcement.getType()).addProperties(properties).build());
  85. announcer.forceAnnounce();
  86. }
  87. private static ServiceAnnouncement getPrestoAnnouncement(Set<ServiceAnnouncement> announcements)
  88. {
  89. for (ServiceAnnouncement announcement : announcements) {
  90. if (announcement.getType().equals("presto")) {
  91. return announcement;
  92. }
  93. }
  94. throw new RuntimeException("Presto announcement not found: " + announcements);
  95. }
  96. }

3、测试RESTful接口

这步需要安装需要的插件,检查插件是否安装。使用postman类似的东西,发送application/json的PUT请求到http://localhost:8080/v1/catalog/,body为

  1. {
  2. "catalogName": "test",
  3. "connectorName": "mysql",
  4. "properties": {
  5. "connection-url":"jdbc:mysql://localhost:3306",
  6. "connection-user":"root",
  7. "connection-password":"root"
  8. }
  9. }

我们可以看到控制台,重新输出了connector的信息:

  1. 2018-08-19T14:09:03.502+0800 INFO main com.facebook.presto.server.PrestoServer ======== SERVER STARTED ========
  2. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap PROPERTY DEFAULT RUNTIME DESCRIPTION
  3. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap connection-password [REDACTED] [REDACTED]
  4. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap connection-url null jdbc:mysql://localhost:3306
  5. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap connection-user null root
  6. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap allow-drop-table false false Allow connector to drop tables
  7. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap mysql.auto-reconnect true true
  8. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap mysql.connection-timeout 10.00s 10.00s
  9. 2018-08-19T14:09:23.496+0800 INFO http-worker-133 Bootstrap mysql.max-reconnects 3 3
  10. 2018-08-19T14:09:23.876+0800 INFO http-worker-133 io.airlift.bootstrap.LifeCycleManager Life cycle starting...
  11. 2018-08-19T14:09:23.877+0800 INFO http-worker-133 io.airlift.bootstrap.LifeCycleManager Life cycle startup complete. System ready.

接下来要怎么利用,要看大家的脑洞了:)?
参考文献: Presto技术内幕